Given the number of vulnerabilities and exposures that are revealed every day, Mike Carlson, Bryley’s Chief Technology Officer, and Garin Livingstone, Director of Operations, agreed to be interviewed to walk me through the process they go through to keep computer systems updated. Mike has deep experience with local- and wide-area network design and implementation, and is a Microsoft Certified Systems Engineer™. Garin is a Microsoft Certified Technician™ and holds an A+ Certification, and has expertise in operations and technology.
—
Q: How does Bryley determine what gets patched and when?
MC: Microsoft releases their patches the second Tuesday of every month. We hold off approving those patches for at least a week to ten days. Then we will start to patch if there are no issues with the patch. Servers are patched on the weekend. Workstations tend to be Tuesday nights, although that depends on the industry and the specific client’s needs. We look at other installed software, as well, to inform us when to patch and reboot client’s systems. We’re in the process of implementing a tool to do this more efficiently.
GL: A CPA business, for example, has their really busy time and then they have their less busy time when they usually don’t have to work three shifts. So usually it’s very easy to pick a day during the week to say, ‘hey we’re going to reboot all your computers at midnight’ and they’re usually perfectly happy with that.
Servers we’ll update on Sunday at three o’clock in the morning — everybody’s sleeping. It gets a little bit different and tricky when you have three shifts, like for manufacturing, for example, because they can’t have any downtime whatsoever. And if they have X number of production servers, sometimes we need to coordinate a little bit more closely with the individual client when those circumstances come up.
It’s also possible the client will pay for the redundancy of having two servers running the same operating system with the same programs — basically mirroring each other, so you can shut one down, do the maintenance on it and bring it up, and then do the maintenance on the other one — so that the business is never down. It’s like Office 365 — they have about 55 data centers all over the world replicating the same data, so that when they do maintenance on one server, nobody’s the wiser.
MC: But that gets very costly very quickly. A lot of the times the client just wants to be able to control when their downtime is. So we’ll apply the patch and tell them, ‘this server’s been updated, you need to restart it within the next few days.’ And if that doesn’t happen, we will follow up with them.
Q: Doesn’t Microsoft sometimes release a patch, and something they didn’t catch in testing ends up causing major problems?
MC: Those things are usually detected pretty quickly, in the first few days. Microsoft will tend to pull back a patch that’s really bad. And that’s actually what’s behind our one week delay. By waiting that week to ten days, our clients are spared the problems.
More often what happens is patches cause issues for some people, but not others. Part of the Bryley patch approval process is using software that evaluates patches and client’s systems and lets us determine whether or when a patch gets installed. Also part of our approval process is looking at both the trade publications and Microsoft’s own documentation on what issues they’ve found. So we will review very closely. I am always reading Microsoft bulletins and trade news media to keep up on vulnerabilities and issues with patches. It’s especially important with security software, like firewalls — if there is a major issue, we will update it, and otherwise keep it updated on a regular basis.
Q: Is a week really enough time to shake out all the conflicts that can happen with a patch?
MC: Conflicts tend to pop up very quickly. If Microsoft isn’t publicizing that there’s a problem, the ERP [Enterprise Resource Planning] vendor will publicize that there’s a problem, telling their clients, ‘don’t install this,’ and telling the trade media to tell people not to install this. We really find that that week’s delay, and at certain times we’ve extended it to two weeks, seems to shake out most of the problems. We don’t encounter a lot of issues where we had a patch, waited that week and then seen conflicts later.
Q: Is Microsoft changing to prevent your ability to delay the patches?
MC: They moved in that direction in the home versions. I don’t believe with the Home Edition of Windows 10, you can delay updates for any significant time anymore. The Professional and Enterprise software you can delay for a time. I don’t see them taking that away from the business side, because they are getting resistance. The IT industry is saying, ‘no, we’re not going to use your product if you don’t let us have a sufficient testing window.’ Especially in enterprise class, people can migrate to a Unix or Linux environment on the server side. So Microsoft seems to be moving toward that model, but they are not forcing us to not be able to delay patches until problems are found and reported.
Q: Do you foresee the Microsoft environment being more like the more-protected, “walled garden” approach Apple prefers?
MC: There are some Home versions of Windows that are similarly tightly controlled. For business class stuff, I think they’d lose the total advantage over Apple, and probably most of their market share, if they insisted on vetting every app before you could install it. That would remove the reason to have a PC with Windows on it. Linux on the desktop hasn’t caught on, but if that’s the only way to install the applications you want, that’s the way people are going to go.
One change Microsoft is making is they are moving to a much more rapid update cycle. They tend to release two major updates to Windows a year — not security patches — I mean feature changes. And you generally can only count on Microsoft to support three versions back. It used to be that they would support multiple versions for years. Now it’s basically the current version and the last two. So you are forced to upgrade. But that’s still a year to a year-and-a-half before you’re forced to upgrade.
And there are times where we will read about an extremely critical patch and we will review and release it, what they call, “out-of-band” [not the normal schedule]. They will release it any time, because there’s a major security hole and we will monitor to test it, and depending on the severity release it outside the typical schedule.
Q: What’s Microsoft’s reasoning for more aggressively forcing users to update?
MC: Microsoft’s rationale is they don’t want to support multiple versions of software. It is more cost to them because if they’re supporting five different versions of the software, they have to test all five versions. Fifteen years ago every service pack of every operating system was supported. But the new rapid update pace for Windows 10 is really based on what their competitors are doing. Apple does that, and Apple really kind of was one of the first to say that everybody updates, “we don’t support five versions, we support two versions.’ So Microsoft saw their competitor as being innovative by forcing this. And that’s where they want to go, because it reduces costs to maintain all these different software versions.
Q: Do you find Bryley clients use Home versions of Windows that lock out any kind of patching oversight?
GL: Predominantly all of our clients would have a contract with us where we would support all of their servers and endpoints. We recommend that they always get business class hardware with the Pro or Enterprise operating systems, so from the get-go we can manage the patches properly. If they have a Home operating system, they’re not even going to be able to join their work domain, so we usually end up upgrading that software. Every once in a while you have a home computer that someone is using to remote into a work computer, but we prefer not to support those cases. We usually encourage clients to get the appropriate hardware and software, so that we can manage patches properly for them.
—
Because of the frequency of exploits and the complexities of networked systems, patching rarely gets the attention it needs. Proper management is critical to systems security. Bryley has the resources and experience to help you stay on top of patch updates, safeguarding your data from security threats. Call the Bryley team at 978-562-6077 or email ITExperts@bryley.com
