Playing with IPMIs and iLOMs, some impressions of.

Over the last few months I’ve had the dubious pleasure of playing with Integrated Lights-Out Management systems (ILOs, iLOMs and iDRACs).

I’ll admit that up until 3-4 years ago I used to think of ILOMs and IPMIs as little more than huge gaping security holes (From a similar POV as Bonkoski et al (https://jhalderm.com/pub/papers/ipmi-woot13.pdf), and similarly the US CERT advisory TA13-207A (https://www.us-cert.gov/ncas/alerts/TA13-207A)).  In point of fact, I’ve actually used Metasploit’s IPMI-related plugins and a few other tools to identify, map and track the use of the above in a couple of companies I worked in.

And I can tell you that unless you fence them in through layers and layers of DMZs, they are bloody dangerous. Because they give the intruder complete and total control of all the hardware (and I do mean all) in your bare-metal infrastructure. You can shut your server down, you can start it back up, restart it, reconfigure interfaces, get a TTY console, pretty much own the system at the hardware level.

NetSec (justified) paranoia aside, though, they are a very important resource in a SysAdmin’s belt, especially if the data-centre is on the other side of the country and you have to either travel like crazy throughout the night to get to it, or submit an SR for an engineer to go do some maintenance. And since the engineer won’t have the authority to mess around with the OS, you’ll need to keep an eye on the system and help them shut down the system and bring it back up safely, otherwise your hard drive change will turn into a P1 incident of the most painful sort.

There are as many different IPMI implementations out there as there are hardware manufacturers, and while they nowadays offer both a number of connectivity options, from serial console to telnet, to ssh to HTTPS web interfaces (which tend to be a PITA to sort SSL certificates for, believe me). But unless you work in an SME (which I am not) or are damn lucky, you will usually have to ssh/telnet (yes, they still use telnet!) to one, or even get the engineer to plug in a serial cable and get access via putty/minicom.

Now, the commands for doing things in an iLO/iLOM/ALOM/etc depend on what the hardware are, and they are usually buried in the hardware manufacturers’ documentation.

What I’ve found as a huge huge help is the cheatsheets that exist out there, for example the GeekDiary ones, found in http://thegeekdiary.com/most-commonly-used-ilom-commands-cheat-sheet/. They are invaluable for them nights where you’re on-call and you have to do something dangerous without having to shift through hardware manufacturers’ documentation sites and KBs.

Now, the thing to remember here, is that when doing anything more complicated that getting a console to the OS, you need to be aware of the state of the OS itself.

For example, if you have to do a hard-shutdown or cold-reboot of a system, its so, so damn easy to think you can just do a stop –force /SYS and then a start /SYS and still expect the OS to start nicely.

Reality is, however, somewhat different. A system crash or kernel panic can have one or more underlying causes, especially in systems with an uptime measured in years rather than weeks. And in such a case the problem will either be made worse by a hard-shutdown/cold-reboot or will result in a corrupt OS well beyond the abilities of an fsck to sort out.

A well-maintained OS on the other hand will, more often than not, come back just fine, or with just minor issues that an fsck will be able to fix, but again that’s debatable.

So, if you start playing with IPMI-related systems, remember the “don’t try this at home, kids!” warning older TV show presenters used to say, and be careful what you do.

That’s all for now, folks, so g’night, and stay safe!

Posted in Network Engineering, SysAdmin Tips & Tricks | Leave a comment

The line in the sand: Thoughts on baselines

First post in a long, long, long time. Too much has happened all of which is downright weird. Still new job, even more fun and exciting, this time back to working with bare-metal (as well as some public and private cloud) and virtualised servers. That’s for a later post, though.

This one’s about what happens when one starts working in companies with bare-metal and virtualised environments in various states of maintenance. Specifically, it is the part that comes just after one starts getting an idea of what does what, how and why, and its called “establishing a baseline”.

Wikipedia defines a baseline as “..an agreed description of the attributes of a product, at a point in time, which serves as a basis for defining change…A “change” is a movement from this baseline state to a next state. The identification of significant changes from the baseline state is the central purpose of baseline identification” https://en.wikipedia.org/wiki/Baseline_(configuration_management)

So, defining a baseline, therefore, involves finding and agreeing on the most optimal configuration of something, and documenting it as the default configuration. Any changes from this default are either incorporated into that default when they further optimise or add to it without any loss of its default functionality, or rejected when they subtract from it or when they compromise it (or both).

The benefits of defining a baseline are immediately obvious: Any changes from the defaults are immediately (or with little enough effort) visible, any errors traceable and most often easily correctable by systematically rolling back your changes until you spot the problem and correct it or by reverting to the default configuration.

In traditional System Administration, one can define the baseline configuration as the state in which your application/service is running smoothly for a pre-defined period of time, a pre-defined number of users, a pre-defined workload, a pre-defined OS & Hardware configuration and a pre-defined I/O & Network throughtput.

Which is why we NEED to know and understand the system/service/application well enough, and which is where a System Administrator needs to gather metrics, run tests and simulations, see what they have to work with (in terms of resources and time allotted) and, perhaps most importantly, actually talk to the users of that system (be it customers, colleagues or both) and management. That will allow the System Administrator to understand what people expect from it and how they expect it to perform, what its importance to the company is, and what the company reasonably expects to pay to maintain it.

In an ideal world, with no restrictions imposed by funding, customer & management expectations, one can take time to bring the system/service down, untangle the mess they inherited, put it back together and party. In the real world, where we get paid for our work and we require nourishment and rest, this takes skill, knowledge, management buy-in, a line manager that cares about their job and is willing to trust you, and loads and loads of baby-step changes preceded by a series of thinking-cap-on sessions, in between breaks, firefighting and sometimes even in bed, in the small amount of clear-thinking time we get between wakefulness and sleep.

You can start bottom-up or top-down. You can start with the hardware and go up to the application, or the other way around. If you got the time, I’ve found its easier in the long-term to start with the hardware and move upwards.

Take your time to pick a good enough hardware kit, with a good enough set of support contracts with a good replacement & service policy. Ensure your network infrastructure is fit for purpose and can take the beating under heavy load. Make sure your OS is up to date, your repos official, your packages signed, your OS config sane, your yum excludes stable yet reasonable. Ensure your app/service configs are sane, and your init and shutdown scripts are, too.

But in a pinch (defined as “you have the proverbial ‘balls’, a good and valid support contract with all vendors involved, and a good and tested set of backups & backup & restore policy for it”) you can start with stabilising the application/service, using only by-the-book configuration and run-time options, setting sane and stable, and move down. Just make sure your baby-steps are (at the very least) human-baby (baby-turtle, even, if you can stretch the time and budget enough) not 7-foot-alien-baby steps!

That should hopefully get you started on the mind-set you will need to bring one service/app to a baseline. Rinse and repeat for the rest. I did mention this WILL take time, didn’t I? 🙂

It does require management-buy-in, and you will have to sweat, but I’ve had visible results in weeks. And the managers that did listen (reluctantly at first) and did allow it, liked it. A lot!

That’s all for now, folks, time for this young man to go to bed and get some rest before his on-call Friday’s next call-out! Stay tuned for more OOH wisdom! 🙂

Posted in SysAdmin Tips & Tricks, Thoughts, Uncategorized | Tagged , , , , , , , | Comments Off on The line in the sand: Thoughts on baselines

New thoughts for a new start of a new journey

 

Yes, I decided to taking another shot at blogging, after basically more than 5 years of not writing anything for anything.

I’ve been through a lot, over the last 5+ years. From PhD Research Student/Academic & DF guy (in Greece) to Penetration Tester, to Linux Engineer, to System Administrator, to Live Services Specialist (which, amusingly, is basically a System Administrator in a DevOps-heavy environment).  A journey that took me from one side of the UK (Sunderland) to the other (London) and back (Stanley, Durham).

I’ve also weathered perhaps the most silly, pointless and downright daft economic depression of my generation, and that over two countries (Greek, remember?). And, believe me, it was hell in oh-so-many-different ways!!

And now, I’m back where I started my journey, back in the North East of the UK, working in a job that’s actually quite fulfilling, teaching me loads of stuff, and allowing me to play with an insane (trully!) amount of tech!! Its not what I initially set out to do, true. But its both the sum total of all my past jobs, and more than that. Yeah, an FTSE100 multinational company does that to a person, exposes them to tech that most SMEs would wish in their fondest dreams. And, while I do still struggle (truth be told) to get used to the transition, I am having quite some fun! And, equally importantly, I have cool, learned and knowledgeable colleagues to go with the new job!! As I said, fun!!!

So, what’s in store for this blog? When I first set out to play at being a blogger, I envisioned it having only netsec and digital forensics posts. Not anymore, not by a long shot.

It will have NetSec and DF treatises and rants, but it will also contain bits and pieces of a hell of a lot more, from mini-reviews (read: rants) of tech I’m playing with, to thoughts on a variety of matters NetSec/DevOps/Coding/SysAdmin I ran across, and still do in my present job, even to thoughts of life/politics/vaping (yes, I’ve quit smoking almost 2 years ago, since I started vaping (4 years ago)). 1-2 recipes may even find their way here, who knows? Because I plan on having shitloads of fun with this blog from now on!

So, when should you expect to see new things from me here? I’ll aim for once a week (Sunday most likely), but hopefully I’ll get back into it full swing so hopefully you’ll get far more frequent updates. 🙂

 

Until then, I bid you good night and may your journey to work tomorrow be pleasant! 🙂

Posted in Life, Thoughts | Comments Off on New thoughts for a new start of a new journey

Adventures in Arch Linux Land

A few friends have forever been trying to get me to change distros, namely to anything but Xubuntu which has been my mainstay for the last 4-5 years (prefferably Arch). I’ve played a bit with Debian and its Kali Linux spin-off (which no offense, OffensiveSecurity, IS actually a viable (and pretty decent) primary-desktop distro!).

Yes, for a while I did change to Kali Linux, mostly to try out some CUDA password cracking with pyrit etc., but the distro (tools aside) felt a bit flimsy a spin-off after a week of heavy use, although I do understand this is their 1st version.

Another mate (you know who you are, hehehe) has been getting me to start using Gentoo again. His reasons were very sound and very good but, tbh, the prospect of waiting around for portage to compile stuff after a 13 hour (commute+work+commute) day is more than I can bear at the moment, even more so having to re-compile stuff when updating the system or when fixing stuff.

While all this has been going on, one of my colleagues also told me about Arch and strongly suggested I try it out on my production system, and after my xubuntu setup got messed up for the Nth time due to some PPA inconsistencies, I gave it an honest try.

And I can’t say I’ve regretted it! Its a rolling-relase distro, much like Gentoo, but without the necessary hassle of re-/compiling stuff all the time. Its fast, its very flexible, not difficult at all to work with and its been a pure pleasure to work with so far. And the pacman + yaourt combo seems to work lovely for all the tools I require for work.

Only niggles:

– The new Predictable Network Names convention: WTF???!! I mean, why? Change for the sake of change is both pointless and stupid! The Linux world has survived just fine using ethX and wlanY and the if scripts for years, why change it this way, breaking pretty much every tool that DOESN’T use this convention/scheme?

NOTE: ln -s /dev/null /etc/udev/rules.d/80-net-name-slot.rules as root reverts us to the old-school ethX convention.

– The systemctl service control (part of systemd): Why, oh why did you change to this monstrosity? Where’s the KISS in that?

Posted in Thoughts | Comments Off on Adventures in Arch Linux Land

New Job, Old Blog, New Interest in the Old Blog.

Greetings, all.

First of all, yes, the blog is still alive, and so am I. In short, I decided to keep things up and running, and to actually start writing in here again. Hopefully I’ll stick to it. 🙂

Yes, I now have a new job, not as a Penetration Tester but as a Linux Engineer. Back to the old-school art of network administration, then, but with all the knowledge of network security & digital forensics being used and utilised to their limits in a VERY multi-OS workspace. In short, I’m loving it!!

So, be prepared to see blog posts about NetSec & DF knowledge applied to Network Administration, as well as classic Network Admin posts.

 

 

Posted in News | Comments Off on New Job, Old Blog, New Interest in the Old Blog.

Job Title: Pentester, Penetration Tester or what?!

Okay, here’s something you don’t come across every day:

Try answering the question “What do you do for a living?” with “I work as a Pen-tester for <name of company>.” or “I work as a Penetration Tester for <name of company>.”!!! I double-dare you!!

To summarise a long and relatively PG-rated story, the following was what non-IT people (including renting agencies, landlords/ladies, the bank, neighbors, local shop owners, utilities people etc) understood:

  • Pen-tester: I work in a factory testing pens.
  • Penetration Tester: I work in the porn industry (presumably as an actor).
  • Penetration Tester: I work in a condom-production facility.
  • Penetration Tester: I am a filthy pervert and I am putting them on.
  • Penetration Tester & Pen-tester: I am putting them on! There is no such job!!!

In the end, all I needed was to preface my response with “…a Network Security…”, and suddenly there were more “what is that, like?” and “Ohhh, so you do computers, then?” and none of the grief!

In short: LOL!! Ain’t non-IT people presumptuous much?!

Posted in Thoughts | Comments Off on Job Title: Pentester, Penetration Tester or what?!

System.out.println{“Hello world! YET Again!”};

Hello folks (for the 3rd time, lol!)!!!

 

New life, new job, new everything, figured out I should have a new blog too.

 

So, stay tuned, and hopefully we’ll all have fun with my adventures in Penetration Testing land (yes, I’ve turned back to pen-testing for a living, now).

 

And this time I’ll have plenty of time to come up with new stuff to update it on hopefully a regular basis with.

 

Cheers,

DarkSYN

Posted in News | Comments Off on System.out.println{“Hello world! YET Again!”};