Just looking at the bandwidth graphs and figured I’d share them with everybody. Recently there was an exploit that allowed random networks to participate in widespread DDOS attacks causing problems across the internet. It seemed like watching our bandwidth for a few weeks through a school vacation would be a good way to identify unusual traffic. Here are a couple quotes from www.us-cert.gov regarding Network Time Protocol Amplification Attacks:
“Recently, certain UDP protocols have been found to have particular responses to certain commands that are much larger than the initial request. Where before, attackers were limited linearly by the number of packets directly sent to the target to conduct a DoS attack, now a single packet can generate tens or hundreds of times the bandwidth in its response. This is called an amplification attack, and when combined with a reflective DoS attack on a large scale it makes it relatively easy to conduct DDoS attacks.”
“US-CERT has issued an advisory that warns enterprises about distributed denial of service attacks flooding networks with massive amounts of UDP traffic using publicly available network time protocol (NTP) servers. Known as NTP amplification attacks, hackers are exploiting something known as the monlist feature in NTP servers, also known as MON_GETLIST, which returns the IP address of the last 600 machines interacting with an NTP server. Monlists is a classic set-and-forget feature and is used generally to sync clocks between servers and computers. The protocol is vulnerable to hackers making forged REQ_MON_GETLIST requests enabling traffic amplification.”
Here’s our usage for the past four weeks, any comments/observations or questions are welcome:
Total Bandwidth for City/Schools
Notice a drop in traffic during February vacation week. Blue is download traffic and red is upload. The lighter blue line represents baseline data; a calculated expectation of data usage during this time period based on history.
Here red is download traffic and blue is upload. City traffic makes up a fairly small percentage of our total bandwidth usage as would be expected.
This reading was taken inside the school web filter and includes cache hits on that device and denial traffic (blue is download, red is upload). Peak here was 80.97Mbps as compared to a 55.6Mbps peak on our total connection during the same time period. It would appear from this graph that we do achieve a significant benefit from caching enabled on this filter.
Red down, blue up. We’ve defined QoS on this network to limit bandwidth to 15Mbps at the firewall; this connection also takes advantage of caching on the web filter and data is representative of that allowing peak transfer of 23.85Mbps though 15Mbps is the limit at our actual internet connection.
Based on these observations traffic appears normal over the last 4 weeks. Periodic audits such as these help us identify anomalies and determine our overall need for services as demands on our network change.
As promised during my presentation at the 2013 Massachusetts Digital Government Summit here is a copy of the slides I used for the “Shared Services – What Works?” session on October 21. I apologize for the delay in getting this published.
We have a private cloud environment that supports police, fire, education and general government. In this environment we run mostly Windows servers (AD, DNS, DHCP and file/print) that support several SQL databases/apps, Exchange, and virtual desktops. We also have a few LAMP servers. We have an off-site DR location that consists of NetApp storage running SnapMirror to replicate data between our primary DC and the DR site (known internally and affectionately as LV426). We have 3 distinct tenants in our cloud: the City of Melrose, the Melrose Public Schools and the Town of Essex.
At this time we are at an operational/conceptual/planning crossroads – do we spend capital to grow our DR site into a full blown second data center (with all of the associated costs) or do we build that second DC in the cloud. The economics seem to favor a cloud based solution but we needed to see it in action and determine if it would work as envisioned and if performance would be what we required.
For the last several weeks we have been working with a vCloud-based service provider (on a trial basis) to build an off-site, cloud-based data center. The goals of the project were the following:
- Develop a proof of concept around using cloud service providers for redundant DC services (rather than build our own physical DC)
- Evaluate different types of workloads in cloud environments – Hosted Exchange, Exchange DAGS and virtual desktops
- Gain experience in using vCloud-based solutions offered by SP’s
In this and follow-up posts we’ll take a closer look at each of these goals and the setup of the cloud-based DC.
The SP Cloud Setup
We partnered with a vCloud powered service provider (SP) and were given a vPC with 30GHz vCPU, 120GB of memory and 6TB of storage and given a management console for creating, managing and assigning resources to vDC’s. The management interface was easy to use and understand. We were able to quickly and intuitively build and manage vDC’s. We provisioned a small vDC with 10Ghz of vCPU, 24GB of memory and 1TB of storage.
To avoid confusion I’d like to take a minute and explain the two interfaces we were using. The vPC interface is separate and different from the vDC interface. The vPC interface is pretty simple and is used to provision resources for a vDC and assign admins. Once you have a basic vDC configured, admins access the vDC via a separate link and can begin to configure networks and VM’s. In our case the vPC interface was very easy to use while the vDC interface was less so.
We accessed our vDC and were presented with a vCloud Director interface. This was a little different than what we were used to with other SP’s. The interface, for those who have not used it before, had a learning curve that slowed our deployment. Our staff was not familiar with vCloud Director. We were told that an easier interface would be forthcoming and I can see how that would be a valuable (almost necessary) component. Challenges of training existing staff, dealing with turnover and the subsequent re-training would be hurdles to long-term adoption. An easier interface would ease that pain point.
Within this vDC we built a small Windows environment that included a Windows Server 2012 domain controller, an Exchange server and a Windows 2008 R2 Server running Desktop Experience to support desktops. The three VM’s were deployed in a single vApp. We also configured our vDC network and Edge Gateway in a pretty straightforward setup. We hoped to use this setup to support a small hosted Exchange demo. We uploaded ISO’s and OVF’s to build VM’s without too much difficulty.
We repeated this process for each use-case workload. We decided to separate our use case workloads into separate vApps rather than build separate vDC’s. This made a few things easier for us, such as reducing the number of networks and site-to-site VPN’s that needed to be created, and helped speed along testing. We added resources to the vDC as needed.
The design within this vDC is subject to change and will probably be very different as we proceed along our evaluation. Part of this project is testing different configurations and familiarizing ourselves with the pros and cons of each design.
The next step for us was to develop some baseline metrics on the environment. This presented our first challenge. The vCloud Director interface we were using did not provide much visibility into the performance of our vDC or VM’s. We are used to using the vSphere client and monitoring some basic host and VM performance metrics. Lack of basic VM performance monitoring is a major drawback and I’m hoping to see some of that functionality in the SP’s promised interface.
Once we completed our base DC configuration we had a vDC running 3 vApps, one for each of our use-case workloads with a site-to-site VPN connecting only the appropriate vApp’s back to our on-premise DC environment. Initially we only needed to connect the vApp running virtual desktops back to our data center. The hosted Exchange vApp would be configured to stand alone and serve as a demo environment. The DAGS vApp would be used to test a variety of DAGS scenarios.
With a basic cloud-based data center built and connected to our primary DC via a site-to-site VPN we felt confident that this approach could be a viable solution. Now we needed to test performance.
Workloads in the cloud
For our use case we want to test the following workloads in the service providers cloud:
Hosted Exchange – we want a demo site for showcasing hosted Exchange as a solution for potential regional partners.
Exchange DAGS – we want to setup a 2nd vDC and test an Exchange DAGS config across two vDC’s. As a follow-up we want to deploy DAGS for our production Exchange environment.
Virtual desktops – we will deploy nComputing L300’s within our internal network that are using VM’s (running vSpace) based in the vDC.
In a follow-up post we will describe some of our performance findings along with some more detailed information on the above workloads. We’ll also share general observations from our experience in using a hybrid cloud with a vCloud powered service provider. Thanks for reading.
Google has released a managed enterprise version of Chrome for deployment in business environments and paired a set of admin templates for management via group policy. We’ve been struggling with Internet Explorer compatibility issues and the inability to upgrade IE beyond version 8 on Windows XP clients has become a problem. A percentage of the hardware deployed in our
environment is not a candidate for Windows 7 upgrade so we decided to try deploying Chrome and manage it by pushing network policies just like we do for IE. If you’re familiar with group policy admin templates the setup is fairly straightforward and Google allows for a ton of configuration to be pushed to the browser. An MSI installer package allows administrators to deploy the enterprise installation silently over the network as well. The MSI and policy templates are available for download here.
There didn’t seem to be much instruction for those unfamiliar with adm and admx templates so here are a few quick steps.
After you download the files from the above link and extract them, you’re going to want to copy some of them to the SYSVOL on your domain controller. Over the network you can browse to \\domain_controller\SYSVOL\domain_name\Policies\PolicyDefinitions (on a Windows 2008 DC, replacing portions in grey italics with names from your own environment).
I recommend using the ADMX template files as they are written in XML and support new group policy settings for Windows Vista and newer. Copy the chrome.admx file into the root of the policy definitions folder.
In our environment we manage Microsoft Office settings this same way so a number of admx files already exist here and we just add chrome.admx to the list, but this folder may appear empty otherwise or you may have to create it. If you do not already see an en-us folder here, copy it from the group of files extracted from the google policy templates archive. If you already have this folder, you’ll just need to copy chrome.adml from the extracted files into this folder.
With these files in place your group policy management console will automatically recognize administrative templates for Google Chrome and allow you to create and assign policies just like any other windows setting over the network. Create a new GPO in GPMC and you’ll see all of these new settings available under administrative templates!
IDG’s CIO magazine announced Friday that the City of Melrose will be a recipient of the 2012 CIO 100. The 25th annual award program recognizes organizations around the world that exemplify the highest level of operational and strategic excellence in information technology. See announcement here.
In addition, Melrose Chief Information Officer Jorge Pazos is a featured blogger for Forbes Magazine, writing about the city’s regional IT initiative. In a first-of-its-kind agreement, Melrose provides computer services for the Town of Essex.
“The CIO award is a much-deserved tribute to our Information Technology department, which has consistently demonstrated the sort of forward thinking that is so important in our rapidly changing world,” said Mayor Rob Dolan. “They have not only made our city’s technology infrastructure more efficient than ever before, they have also extended their services to other cities and towns. This is the wave of the future, and I am proud that Melrose is at the forefront—and that we will be bringing our message to a wider audience via Forbes Magazine.”
Executives from the winning companies will be recognized at the CIO 100 Symposium & Awards Ceremony, to be held Tuesday evening, Aug. 21st at the Terranea Resort in Rancho Palos Verdes, Calif.
About the CIO 100
The recipients of this year’s CIO 100 award were selected through a three-step process. First, companies filled out an online application form detailing their innovative IT and business initiatives. Next, a team of judges reviewed the applications in depth, looking for unique practices and substantial results. Finally, CIO editors reviewed the judges’ recommendations and voted on the final 100.
Complete coverage of the 2012 CIO 100 awards will be online at www.cio.com on August 1, 2012 and in the August 1st issue of CIO magazine.
About CIO Magazine
CIO produces award-winning content and community resources for information technology executives and leaders thriving and prospering in this fast-paced era of business, as well as creates opportunities for information technology and consumer marketers to reach them. The CIO portfolio includes CIO.com, CIO magazine (launched in 1987), CIO Executive Programs and the CIO Executive Council. CIO properties provide business technology leaders with analysis and insight on information technology trends and a keen understanding of IT’s role in achieving business goals. The U.S. edition of the magazine and website are recipients of more than 200 awards to date, including the American Society of Business Publication Editor’s Top B-to-B Magazine since 2000 and two Grand Neals from the Jesse H. Neal National Business Journalism Awards. CIO websites and printed publications appear in more than 25 countries, including Australia, Canada, Finland, India and Sweden. CIO Executive Programs—a series of face-to-face conferences including the CIO 100 Awards & Symposium™—provide educational and networking opportunities for pre-qualified corporate and government leaders. The CIO Executive Council is a professional organization of CIOs created to serve as an unbiased and trusted peer advisory group. CIO is published by IDG Enterprise, a subsidiary of International Data Group (IDG), the world’s leading media, events, and research company. Company information is available at www.idgenterprise.com.
This week we have a guest blog post from James Covino, a Melrose High School graduate, current Bentley University student and part-time Melrose IT staff member. Enjoy the post and make sure to follow James on Twitter @jimcovino.
For those of you who missed it, Google finally released its beta of Chrome for Android last week. With its release, Google has promised increased speed and simplicity over the default Android browser, along with tighter integration and syncing with the desktop version of Chrome. That said, it is important to note that the beta is available only to the 1%. That is, the 1% of Android users running Ice Cream Sandwich. Furthermore, Chrome has no support for Flash and there are no plans for it in the future. This is a logical step in the evolution of the internet away from flash and towards HTML5, but worth pointing out to the few who might miss it.
Anyway, I spent the past week with the Chrome Beta to see how it stacks up…
The default Ice Cream Sandwich browser has always been fast (garnering its fair share of critical praise), but Chrome is, well, Chrome fast. Chrome for Android seems to have just as much pep in its step as its big brother over on the desktop. This increased speed shines through both in numbers and its more streamlined interface. Google’s implementation of the omnibox and pre-loading search results simply make it feel faster. That said, Chrome’s speed is still tangible, scoring 1865.1ms on SunSpider compared to the default browser’s 2014.4ms. In my tests Chrome managed to consistently load pages 2-3 seconds fast than the default browser on average. No doubt about it, just as it is on the desktop, speed is Chrome’s strong suite in Android.
SIMPLICITY & INTERFACE
Chrome on Android is simple. Dead simple.
The new tabbed browsing interface can best be described as a deck of cards. Tabs show up as conveniently oversized previews that can be effortlessly flipped through or pulled apart to get a better view of a specific page. The interface makes solid use of multi-touch and feels very natural.
Opening a new tab is an improvement over the default browser as well. New tabs now open directly to a menu displaying your most recent websites, your bookmarks, and websites open on your desktop (more on this later). The overflow menu is also improved, offering direct links to new tab options and forward/back buttons. These are solid examples of small changes made by Google that ultimately just makes life easier and contributes to the perceived speediness of Chrome.
While trying to make Chrome simpler however, Google omitted some key features that make the default browser great. First and foremost is the option to automatically request the desktop version of websites. This has been one of my personal favorite features in the Ice Cream Sandwich browser, accessible directly from the overflow menu. It is an absolute mystery to me as to why Google took this away. Google also removed the option to save pages for offline reading, partially replacing it with the option to send pages from your desktop to your phone for later reading. Finally, perhaps in its attempt to streamline menus, Google has also removed the option of being able to view your browsing history on your mobile device, not too big of a deal thanks to the functionality of the omnibox, but worth pointing out.
Google has also introduced a new feature called Link Preview, which detects when you are trying to select from a list of hyperlinks and pop ups with a magnified window to make sure you pick the right one. On paper this sounds great, finally, an end to the nightmare of trying to pick out a miniscule link on your smartphone, akin to playing the world’s hardest game of operation. But unfortunately it just does not work that well. In my experience, it worked less than half of the time. That said, Chrome is still an beta and it would be great to see this feature improved upon in the future.
Overall, the new interface and improved simplicity of Chrome make a solid foundation that is hopefully built upon more as it comes out of beta.
Desktop Integration and Syncing
This is what makes Chrome more than just a revision of the Ice Cream Sandwich browser. Google calls it, “your Chrome, on all of your devices,” and that is exactly what it is. Not only do all of your bookmarks sync, along with your omnibox history, but you can access your open tabs on any other instance of Chrome you have running. This is a nice feature that really brings home the integration between your desktop and mobile versions of Chrome.
Even in its beta stage it is pretty clear that Chrome for Android is a winner. I have not encountered any crashes or hiccups after moderately heavy use over the course of the past week (although considering it is exclusive to Ice Cream Sandwich which very few devices are running right now, this does not mean a whole lot) and speed has been consistent. Google has expressed its intentions to eventually make Chrome the default browser in Ice Cream Sandwich once it comes out of beta. Knowing Google, that could take a while, but it seems a lot like Chrome for Android is just about ready for primetime as it is. My only personal gripes are the inability to automatically request the desktop version of sites and the lackluster ability of Link Preview. Others may also be turned off by the lack of support for Flash, but that is simply a growing pain of the internet. All in all, Chrome is nothing revolutionary, but offers what is arguably the best mobile browsing experience available today, particularly for users of the desktop version as well. After the past week it has already replaced the default browser for me, and if you are running Ice Cream Sandwich as well, I highly recommend it.
As part of our process in providing IT services to Essex we had to establish a site-to-site connection between the two facilities and insure that there was connectivity of sufficient bandwidth and quality. In order to do this without the additional expense of MPLS or some similar service we deployed two WAN aggregation devices (one at each location) and established a site-to-site VPN. During the next several weeks we will be testing the quality of this connection to determine if it will be sufficient to support our plans moving forward. Preliminary monitoring is promising and we hope to post some detailed stats on this as they become available.
On October 27, 2011 the Melrose Board of Alderman voted to approve Order Number 12-44. A similar order was approved earlier by the Town of Essex Board of Selectman. This order approved an agreement between the City of Melrose and the Town of Essex to create a regional IT partnership, the first of its kind in the Commonwealth of Massachusetts. Per the agreement the City of Melrose will slowly migrate all datacenter operations from Town of Essex facilities to a hosted environment on City of Melrose infrastructure. Eventually Essex will not have any server or storage infrastructure on premise and all of those functions will be served remotely from the Melrose datacenter. During late 2009/early 2010 Melrose built a new datacenter based on FlexPod. The intent was to consolidate all datacenter operations for government and education and then leverage that infrastructure for regional partnerships. We plan on documenting our progress here on our blog.
We recently added some memory to our Cisco UCS B250 blades. We were careful to order the correct memory modules (they come in pairs) and thought we understood the installation instructions. According to the documentation on the Cisco site we needed to install the matched pairs into channel slot number (0,1) and (4,5) for a 4 DIMM configuration. The 2 DIMM config stated that the channel slot numbers should be (0,1). So of course we expected to find 2 DIMM’s installed in channel slot (0,1) and that we would add 2 new DIMM’s into channel slots (4,5). That would be a totally incorrect assumption. Upon opening the case we found 2 DIMM’s installed in channel slots (0,4). This is how it shipped from Cisco. You can see in the picture that Channel E is configured this way, the rest were as well. So we had a choice to make; Option A – install the matched pair of DIMM’s into channel slots (1,5) or; Option B – move the existing DIMM from slot 4 to slot 1 and install the 2 new DIMM’s into slots (4,5). We went with Option A. Our reasoning was that the UCS was working under the current config with no memory errors and, despite not agreeing with the documentation, was the least disruptive option. Option A did not work, in fact the server wouldn’t even boot. So we opened the case again and installed the DIMM’s according to the documentation (Option B above) even though this meant changing the configuration that shipped from Cisco. The server booted correctly, registered the memory and has no errors. The moral of the story, follow the documentation, even if the manufacturer doesn’t.
When the first UCS app came out for android, there were a couple reasons I didn’t rush to download it right away. From my experience in the regular Java management console I wasn’t sure it would scale to the form factor of a phone very well and I honestly don’t spend a whole lot of time there in the first place. In our environment, once the initial setup was finished things remained fairly automated and any in-depth management of UCS has generally been restricted to projects like hardware upgrades or expansions. I thought the app would probably be more suited to an android tablet and I’ve had my eye on the Cius for a while now so I kind of planned on testing that hardware and this software at the same time. Eventually I decided to give the app a try on my Droid X and realized it was designed a little differently than I expected. The aim of this software is more about insight than management which actually works out well for both the form factor and use cases of a typical smartphone. One of my own shortcomings regarding UCS is that I don’t get into the manager as often as I should to check events and warnings on the system. The UCSand app highlights the value of Cisco’s XML-API in allowing a program to pull targeted data out of the UCS Manager without requiring an individual to navigate through extra configurations or information at the time.
The app is super-fast and it is really nice to have answers to most of the common questions about your servers like “How much memory is in that blade?” or “Which hosts were the ones with Nehalem Processors?” right on your phone. I’ve also checked the event logs on UCS more times today than in the last few months combined; it becomes one of those things you do on your phone while waiting for meetings to start or lunch to end. I really like the app and immediately thought of it as another of those logical, value-added features that convinced us to implement UCS in the first place.You can really see how the trend of lightweight application delivery that has been so successful in the consumer market can and will demonstrate its value in business and the enterprise. Much of systems management is broken up and distributed throughout a team of individuals with specific interests and specializations. We all have certain processes and applications we perform or monitor constantly and the ability to target and deliver access with minimal impact on mobility is a natural goal in improving the efficiency of workflow. It is nice when you can simply query the device in your pocket or under your arm for answers during spontaneous planning or problem solving situations without holding up the whole process. I know I’ll be trending towards increased mobility in the workplace as these technologies mature and develop; it will be exciting to see how manufacturers and customers offer and deploy solutions as the paradigm continues to shift.
If you’re interested in the UCSand app, find it on the Android Market here.