The Day the Cloud Went Down

7:33 AM, August 31, 2019, a power failure hit Amazon Web Services (AWS) US-East-1 datacenter in North Virginia. As expected, the datacenter’s backup generators kicked in.

Then at about 9 AM the generators started failing

The result was 7.5% of Amazon’s data storage units were unavailable for several hours. Some of the affected websites and services included Reddit, Styleseat, Fortnite, Sendhub … and many smaller sites and services. Power was restored, and around noon most of the drives were functioning. But the outage rendered some data unrecoverable. For those that had not backed up their data, the power failure was a disaster.

Did You Feel a Drop? BUDR Pt. 2

Remember the panicked manufacturer with a progression of computer problems that recalled the proverbial “perfect storm”? Today I’m going to sort those problems out, and recommend ways to help keep those things from happening to you. To refresh, here’s what went on with that manufacturer:

The installed version of their ERP (Enterprise Resource Planning software) was too old to receive application maintenance and support. They hadn’t wanted to interrupt their workflow, and so hadn’t upgraded the ERP in more than five years.

Avoiding the Perfect Storm

A nearby manufacturer called Bryley panicked. The problems started with a failed upgrade to their ERP (Enterprise Resource Planning software). Because they hadn’t wanted to interrupt their workflow, the company hadn’t upgraded the ERP in more than five years. But the installed version was now beyond maintenance and support, so they bit the bullet. Only, during the upgrade there was a power fluctuation so that the upgrade did not complete. In fact no computer on the network would boot normally. In addition while trying to resolve the failed upgrade, they found a flaw in their in-house back-ups. Now the business was losing days and money to downtime.

World Backup Day 2019

Celebrated On March 31st

Data is essential to us all.  We rely on it complete even the most basic daily tasks, and yet it is so easy to take for granted.

World Backup Day was established to remind us of the importance of data in our lives, and why we must make a sincere effort to protect it.

Think of all the information you access every day.  Photos, documents, financial records, calendars, reminders, even shopping lists are all stored electronically.  Having all that data right there at your fingertips makes life easier than ever before.  When something goes wrong, however, and that data is gone, even the most basic tasks become nearly impossible.

That is why we encourage you to take a few minutes on March 31st each year to make sure that your data is backed up and secure.  If it is, set aside some time to check and make sure that those backups are working properly and that you have access to them.  If your data is not backed up, mark the 31st in your calendar as the day you will fix that.

For more information on how World Backup Day came about, and how it is celebrated, you should visit their homepage.

Keep Your Electronic Devices Protected – Power Strips vs. Surge Protectors

With all the electronic devices that are used in homes and offices in today’s high tech world, oftentimes there just aren’t enough electrical outlets to plug them all into. Power strips offer a solution which enables a single outlet to handle multiple devices. But if you don’t choose one that also offers surge protection, you could be putting your devices at risk.

The Importance of BU/DR in the Manufacturing Industry

What would happen to your organization if plant production was taken to a halt?  How would you get it back up and running?  Or, could you?

Whether the disaster is caused by mother nature, a human error, a cyber-attack of some sort (and yes, there are many types), it can wreak havoc on your organization – it can even take the company down to its knees.  Each moment of downtime equates to lost dollars and lost customer satisfaction. Manufacturing firms have to effectively ensure that production and distribution is consistent.

Technology is used throughout manufacturing in many ways – to store data, run automated machinery on the plant floor, track inventory and support distribution. Your technology is intertwined with your business processes and if you suddenly weren’t able to use those processes, it could be a catastrophic situation.

A few scenarios of how a disaster can disrupt manufacturing, and what you can proactively avoid it.

Halt in Production.  Complex automated equipment and inventory tracking are just two processes that are severely influenced during a disaster. Do you have a recovery plan in place for a worst-case scenario?  Production logistics may be the most challenging area to recuperate, but having a strong backup and disaster recovery (BU/DR) plan safeguards data and allows for immediate access to mission-critical applications.

Whether your organization experiences a cyber-attack, or even a power outage that shuts down productivity for several hours, all of your applications used to run the automated machinery will not work because the system cannot connect to the network. Depending on the size of your plant(s), you could be facing up to millions of dollars in lost revenue and customer reimbursement.

BU/DR To The Rescue.  If your senior management team turned to a BU/DR expert – like Bryley Systems –  to assess the possible vulnerabilities associated with an outage and developed a proactive plan to recover and access data, your BU/DR provider would be able to access your data and apps to get your operations back up and running with a minimum amount of downtime.

Halt in Distribution.  Downtime is never acceptable when it comes to distribution.  All schedules must be strictly followed to satisfy delivery expectations. Customers don’t care if your warehouse floods.  They want to receive their order on time. Logistics management utilizes computerized tracking and ERP systems to understand how many products are stored and where they are at any given time to enhance product readiness and customer fulfillment.

Imagine this scenario – you work as an IT Director for a large New England pharmaceutical manufacturing company. Your network is more vulnerable to external hacker attempts simply due to the size of your business and the value of your data.   All of a sudden, your systems are corrupted with vicious malware and the entire database is inaccessible. To continue operations at your normal efficiency level and avoid downtime, your backup and recovery disaster plan kicks in to eliminate the malware and restore your plant data to where it was before the attack. Investing in a custom BU/DR plan serves as disaster protection ensuring your ability to move products to their destination.

The key to effective disaster recovery is planning ahead. Partnering with a BU/DR professional to support your critical infrastructure and resources adds additional layers of security and communication. When unexpected disasters strike, your recovery strategy will be there to save the day by restoring your data and reducing your downtime.

 

The Bryley BU/DR process:

  • For on-premise equipment, we deploy a BU/DR appliance onsite to provide local backup-and-restore capability and to speed recovery.
  • We take an encrypted image of your system and copy it to our data center.
  • We stream encrypted, differential changes from your site to our datacenter

Isn’t the survival and security of your manufacturing organization worth the investment of BU/DR?  Our team of experts will help you navigate through this process and implement the most effective BU/DR tailored to your environment and budget.  Contact us at 978-562-6077, or by email at ITExperts@Bryley.com to learn more. We are here to help.

Pop Quiz: How Prepared is Your Company to Recover If Disaster Strikes?

You depend on your IT systems every day, but how dependable are they really?  If your company was subjected to a sudden loss of power could you keep working, or would business stop?  What if the power didn’t come back on for several days, or even a week?  Most importantly, have you already asked yourself these questions, and if so, do you have a written action plan to address them?

If you are at all unsure of how a disaster would impact your business, and how you might recover, here is a great little quiz to help you get the wheels turning.

1. How frequently are your company’s critical systems backed up?  Is it more than once a day?

2. If your company lost power, would your systems keep running without any interruption?

3. In the event of a system failure, could your company’s data be restored to working order quickly?

4. If your company experienced a security breach, do you have a clean set of data backups available and could they be restored in a timely fashion?

5. Does your company have a fail-over site it could revert to if your primary systems become inaccessible?

6. Does your company have a written disaster recovery plan to refer to in case of emergency?  If yes, do your employees know where to find it, and are they trained in implementing it?

If you were able to answer “yes” to these questions, congratulations!  Your company is in relatively good shape in terms of its ability to cope with a disaster.  If you answered “no” to any of the questions on this list, however, it would be a good idea to spend some time putting a plan together to address any gaps in your ability to recover from a disaster.

As a Managed IT provider, Bryley Systems specializes is helping companies plan for disaster and mitigate the risk of a loss of data.  If you think you could use a hand putting together an effective disaster recovery plan, why not give us a call at 978.562.6077.  We are here to help.

No Power? No Problem. Just Plan Ahead…

Natural disasters, severe weather and even cars that knock over electric poles, can all wipe out power sources and cause businesses to lose the ability to communicate through cellphones, landlines and email.  If you are a business owner, being forced to inconveniently and unproductively “wait it out” is usually not an option.  And, power outages can be much more than inconvenient; they can be costly.

A study by the University of Lincoln has concluded that “power cuts will become more regular around the globe as electrical supply becomes increasingly vulnerable and demand for technology continues to grow at an unprecedented rate.”1    And, The Washington Post reports that “the U.S. grid is aging and stretched to capacity. More often the victim of decrepitude than the forces of nature, it is beginning to falter.   Experts fear failures that caused blackouts in New York, Boston and San Diego may become more common as the voracious demand for power continues to grow. They say it will take a multibillion-dollar investment to avoid them.”2

Organizations need to be prepared for a power outage, regardless of the cause. That preparation should be focused on preserving data both inside the office as well as data located on servers stored in an offsite data center.  All of your digital assets, including software, are vulnerable to being wiped out during a power failure.  To protect your business and eliminate that vulnerability, you need to do some prep work to back up your data and implement basic hardware security measures.

Many business owners prefer not to think about this, but safeguarding your assets will allow your organization to at least partially function during an event vs. being at the mercy of the power grid.  The amount of time and money it takes to prepare for a potential power failure is fractional compared to the amount of time and money it would take you to rebuild your empire of digital assets from scratch.  Digital assets get wiped out all the time during power failures.

Back up your data and get your employees on board with performing regular backups of their work. This is the most basic of all requirements for being ready for a power outage, but many people don’t do it, either because they don’t know how or it can seem overwhelming. Not backing up your files is taking a huge risk that everything could be gone in the blink of an eye.

Organizations with larger networks should have backup servers that can continue to distribute data during, or, after an emergency.  Off-site data backup is recommended, and, data may also be saved to the cloud.

Even if you have multiple backup locations for your files and don’t have to worry about losing your digital assets, you still need to be aware of the potential for losing your physical devices like computer hard drives, power sources, and motherboards. It should be noted that solid state drives are not immune to being fried by a power surge.

It is not the power outage that causes damage to your hardware, it’s the power surge that does the damage. A surge protector is one way to prevent damage to computer hardware, but it’s not a guarantee – sometimes they work, sometimes they don’t. How well it works will depend on how much power your particular unit is designed to withstand and if you’ve replaced it.

If your equipment is damaged, the most efficient and cost-effective fix is to replace the unit.  While the hardware may not be expensive, the labor costs alone will often outweigh the price of a new machine.

If you’ve got your data backed up, you probably don’t need to worry much about the cost of replacing your hard drive; most hard drives are fairly inexpensive. Even so, it’s a hassle to replace, so you should protect your hardware as much as possible.

The first line of defense against power outage issues that impact business continuity is on-site power protection. This is a proactive measure that requires planning and implementation before the power outage occurs. To protect data and servers, organizations should have uninterruptible power supplies, or UPS’s, and ideally, an on-site backup power supply, such as a generator. This can ensure that your business suffers no loss of data in the short term, while your continuity plan is being implemented.

In addition to backup power solutions, load-capping software and power distribution units should be considered. For short-term outages, battery backup may be sufficient for communications and VoIP systems. Planning should include identifying and outlining battery specifications and status as well as battery-replacement policies. Emergency battery-powered lighting should be available in multiple areas around your building if you do not have a backup-lighting system.

The cost of providing on-site power for a long period of time can be high, so for outages that last longer than an hour, organizations should have alternative options, such as an off-site location for protecting data and ensuring access.

Securing off-site data backup and disaster recovery solutions, such as Bryley’s Business Continuity, is something every business should have in place before a power outage occurs, and most organizations should take that a step further and move all critical IT infrastructure into the cloud. Data centers provide improved resiliency, reduced power and cooling expenses, and easier infrastructure management, in addition to ensuring continuity during power outages.

A few more tips…

  • Always plug computers and laptops into surge protectors instead of directly into the wall.
  • When your laptop or other digital device is done charging, unplug the charger immediately. This saves your battery from losing charge capacity, and it also makes sure your device won’t get fried if there’s a power surge.
  • Have an uninterruptible power supply (UPS) at every desk and make sure your employees know what to do after a power surge. The UPS may only give them enough power to properly shut their machine down, so they’ll need to act fast.
  • Always unplug your electrical devices during a storm, especially a lightning storm. There is no surge protector in the world that will protect your computer from a lightning strike.
  • Not every organization requires a dedicated generator for backup power because most businesses can tolerate the downtime of a short-term power outage.  Many office buildings have standby generators that companies can rely on.  Be sure to know if you are equipped with a standby generator if your organization requires one.

Remember, having a plan and being prepared is your first line of defense.  If you are concerned about the emergency power outage strategies your business has or some other network issue, please contact us at 844.449.8770 or by email at ITExperts@Bryley.com.

We would be glad to help you assess and mitigate your risks.

References:

  1. https://www.sciencedaily.com
  2. https://www.washingtonpost.com/local/trafficandcommuting/aging-power-grid-on-overload-as-us-demands-more-electricity/2012/08/01/gJQAB5LDQX_story.html?utm_term=.0119ff3e554b

Securing your Organization’s Computer System and Company Data

Cybercrime can have severe consequences for organizations that are victimized. “Cyber criminals want your company’s financial data, customer lists, credit card information, intellectual property, and anything else they can sell. Besides the immediate impact on your organizations’ ability to operate (for example, ransomware can shut you down), there are significant costs associated with a data breach.”1 These costs include:

  • Potential fines from regulatory agencies for failure to protect personal identification data (PI) or personal financial information (for example a credit card number). Some fines have been in million dollar multiples.
  • Loss of customers.
  • Loss of business reputation.
  • Inability to attract new customers or clients.
  • The cost of notifying all persons and businesses that had their data breached.
  • Costs of providing credit monitoring and identity theft protection for all involved.
  • The potential cost of defending a class action lawsuit against your company for failure to adequately protect sensitive information.

The following tips contain some best practices for keeping your organizations’ computer system, programs, and data, safe from prying eyes.

  • Make sure your firewall is on so that intruders cannot access your system from the internet.
  • Install security software and keep it updated.
  • Filter all email for computer viruses.
  • The more popular a program used by your business is, the more appealing it is a target for cyber criminals. Criminals know that many users do not regularly patch their programs against malware infections, so popular programs give them a wider pool for targeting. Make sure that your computer programs are updated as soon as a security patch is released.
  • Exercise caution when using free 3rd party software claiming they check for software updates. Free software may be bundled with malicious software.
  • Instruct employees that they should never open emails from unknown senders since they might be a source of infection to your company’s computer system. Attached malware might be anything from a virus to ransomware.
  • Tell your employees they should check with senders they know if an email has a suspicious attachment. A phone call to the sender can help avoid the consequences of a data breach.
  • Inform employees they should not open websites they are not familiar with and they should make sure the address bar is the site they want to visit. A random click could take them to a website that is just waiting to download malware onto your company computer system.
  • Buy your software from a reputable buyer and never use pirated software as it may be infected with malware.
  • Before using thumb drives and other portable media, scan them with security software to ensure they are virus.

Protecting your company’s data and infrastructure should be a top priority, but you do not need to do it alone.  The Bryley security team has the training and expertise to protect your company’s data and infrastructure. Please contact us at 844.449.8770 or by email at ITExperts@Bryley.com. We’re here for you.

Resources:
1 https://www.us-cert.gov/ncas/tips (US Computer Emergency Readiness Team)
https://www.consumer.ftc.gov/articles/0272-how-keep-your-personal-information-secure (Consumer Information / Federal Trade Commission
https://www.adp.com/who-we-are/data-security-and-privacy/safeguard-your-data/minimize-the-risk-of-computer-viruses-and-malware.aspx (ADP)
www.smallbusinesscomputing.com (Small Business Computing)
https://resource.elq.symantec.com (Symantec)
http://guides.wsj.com/small-business/technology/ (Wall Street Journal)

 

Why Backups are Critical in Fighting Ransomware

Ransomware – usually Crypto Locker and its variants – is a form of cyber-malware based on encryption software that seeks payment (ransom) to undo the damage; when infected, the malware typically encrypts all data files, rendering them useless until the ransom is paid.  (Encryption software scrambles a files’ contents and creates an encryption key, essentially a code used to reverse the process.  Unless you have this key and the encryption software, the files remain unreadable.)

Recently, hospitals and police departments have become victims of ransomware.  Hollywood Presbyterian Medical Center was forced to pay a $17,000 ransom in bitcoin to regain control of their computer systems and after repeated attempts to decrypt their data, the Tewksbury Police Department was forced to pay the $500 ransom.

Other than paying the ransom, which is risky and not recommended since it potentially makes you more of a target in the future, the only way to thwart ransomware is by restoring the corrupted files through a backup that was created before the infection.

A properly planned and implemented backup process is vital since data stored on a network server represents many hours of effort over time, making it impractical and usually impossible to recreate.  A properly functioning, multi-point-in-time backup is necessary to provide restoration under these and other scenarios:

  • A server fails
  • A file is deleted
  • A template is written over
  • An application upgrade fails and must be restored
  • A document is inadvertently changed and saved by a user

A backup should be a complete, recoverable copy of not just data, but the entire server/network environment.  It should have these properties:

  • Sequenced over many days
  • Complete image
  • Offsite storage

If you’re ready to get serious about protecting your business data, select a talented Managed IT Services/Managed Cloud Services company, like Bryley Systems, to help you double-check your IT infrastructure, recommend solutions to eliminate weak links in your security chain, and help you develop an organization-wide policy to help prevent data loss. Please contact us at 978.562.6077 or by email at ITExperts@Bryley.com. We’re here to help.