ericsysmin's DevOps Blog

Automate Everything

Eric Anderson

Ok, almost automate everything. I do recognize that this far into DevOps there are some things you can’t quite automate. But for the most part you can stick to these few rules. If your situation matches any of these, you could probably automate it with a script.

  • The same actions are performed often, more than 3 times.
  • You want to outsource advanced actions to a tier 1 team.
  • You want to allow developers or non-technical people perform the actions.
  • You want a way to make sure the same actions are done whenever something is ran
  • Source control has a huge benefit, I script just about everything, just so I can upload it to source control and roll back a script if there’s a bad change.

Some languages that are really common in DevOps land include Bash, Python, Powershell (my Windows friends), and Ruby, however, I have seen DevOpsers use Java, Perl, Go, and many other languages. Over 6 years and 3 companies, I’ve only seen Bash, Python, Powershell, and Ruby used within those companies.

Continue reading...

Pets vs. Cattle

Eric Anderson
Earlier I had mentioned about Pets vs Cattle, and one of my recent friends asked me what it actually means. Pets vs Cattle is a concept. Recently brought on by the advent of cloud computing and managing massive infrastructures which can be easily automated. Its extremely different than traditional systems administration, which was born from physical machines where you may need to purchase new hardware, build it, then deploy it within your datacenter.

Pets

First you give them a name. Then you installed services on that server, and if the server got corrupted or hardware failed you had to repair the computer, possibly rebuild the hard drive, fix corrupted filesystems, I could go on essentially you nursed it back to health. This is referred to as "taking care of a pet", you take care of it, you update, upgrade, and patch it. Some applications still need this attitude, but many in this decade do not! Now we've changed, and we with RESTful APIs can quickly spin up infrastructure, and build from scratch to a working machine in minutes.

Cattle

Now with cloud computing we are able to do amazing things. I can easily build a "golden image" from a running application, even build that image from scratch and use it in an autoscaled environment (something Netflix has done for years!). Using tools like Ansible, Puppet, Salt, and Chef, you can quickly build new infrastructure that is known to work. You can even use those tools to build a brand new image which can then be used to create 100 servers just with one command line entry. Instead of names these guys get numbers. All servers are essentially identical to each other. If a server dies, you issue a couple API calls (or not if you are using AWS Autoscaling or similar, as it does it for you if you want it to), and now you have replaced that server in your environment. If a cow is ill/dying you kill it, and get another. That's where this analogy originates.
Continue reading...

DevOps Really…..

Eric Anderson

What is DevOps?

DevOps is not one thing, its a plethora of ideas, and practices that have revolutionized Operations and the Release Process. It’s a culture change. DevOps is using development practices and applying them to operations. Then using those methods to automate everything.  It starts to join both Operations and Development together, consider it as a hybrid, I mean it is DevOps, it’s the joining of Development and Operations. DevOps was created to empower developers to quickly integrate and build environments to test code, and allows the operations teams to quickly deploy and monitor applications just as fast as the developers and QA teams can approve it, thus leading to Continuous Integration and Continuous Deployment. DevOps teams that I have started tend to do well with another great practice called Scrum, which are just as important to operations as it is to development, if you don’t believe me try it for a month, then see how effective your team starts becoming. As you may have noticed many of these things are part of Agile practices, if you aren’t familiar hopefully reading and following this blog you will be. DevOps and Agile go hand in hand. DevOps helps reach the goals of Agile effectively, also much easier. You don’t want Operations to be a blocker….that looks bad…wait…it looks really bad.

Some extremely popular concepts come into play here.

  • Scrum
  • Operations Early
  • Infrastructure as Code
  • Automate Everything
  • Test Driven Development for Ops (CircleCI, TravisCI, etc.)
  • Integrated Operations
  • Pets vs. Cattle

Over the next few weeks I will cover all of these, and hopefully more. There’s just way too much to type up in one blog post, and all of it is important (I mean there’s entire books written about this stuff, there’s even series of books written about these practices).

Continue reading...

Providing Static IP in front of AWS ELB (Elastic Load Balancer)

Eric Anderson

So, certain situations can arise (often times security) where you will need static IP addresses in front of your ELBs. Not everyone needs this but a rising number of people are starting to, and I will show you how. The only use case I’ve ran into this is on AWS, not anywhere else. But if it works it works.

Currently there are only two solutions I have found: HAProxy 1.6, and Nginx (goes back to versions farther than 3 years ago). Here’s how

In Nginx:

There is a configuration value called “resolver”, here is an example. It can be used in your  http, server, or location  context.

An in use example:

This will force Nginx to update the DNS for example.com every 10s. In applications I have done we set this to 2s, because it works better for the amount of scaling up and down we do. Since doing this we haven’t seen any failures related to DNS resolution of ELBs.

As a side not, putting it in the http context does have some issues and you may end up with stale DNS data, even with valid=10s, etc.

 

Continue reading...

Socks5 Jump Host Proxy Service

Eric Anderson

So, everyone uses a jump host to help with security. It often causes people to require to configure SSH tunnels, to remotely browse the services and even connect to hosts in another network.

The best information I found was using a command like this:

However you have to enter that at every single boot! There’s no status on it, and that doesn’t help you simply run it in the background even without logging into your jump host.

Solution:

Lets run it as a service! For this instruction I am using CentOS 7, simply because systemd is in the new OS’s, so it’s relevant.

Now you have a no mess, socks5 service running on your host. If you want it to start with boot, just run:

 

Continue reading...