I’ve become increasingly disillusioned with the world of IT. I seem to work harder and harder, both day to day, and with my own training, and the expected range of my job role gets bigger and bigger and my own training more and more inadequate. I plan to resolve this, I’ve been drawing up a plan today to try and cover a wider breadth of skills.
Let’s see how it goes.
Something I’ve been meaning to do for a while is build a firewall appliance. I’ve kept looking over the pfSense and M0n0wall sites and thinking it would be a good thing to learn from. As I might have mentioned previously, my networking knowledge is generally not as good as it might be, and while I have no ambition to try and become a CCIE, it would be useful to have some extra tools in the box. Virtualisation has meant that the role of the server guy has now expanded to include networking and storage, whereas this knowledge would previously have been silo’d into specific teams. The real deep knowledge still lives with the dedicated teams of course, but I just think I need more than I needed before.
So, a week or so ago, I took the plunge and won a Watchguard Firebox x750e on eBay. In the end, it was about £70 delivered. In a previous job many years ago, we bought one of these new and it wasn’t the most intuitive to set up. The licensing for the various features also seemed incredibly expensive on top of the original purchase price. But it’s red and has an LCD status panel, and in geek world, that’s very cool. Even the missus commented it looked nice 🙂 I’d missed a few before because of carelessness and a lack of enthusiasm but I got a really good one in the end, rack ears included, and the box really is as new. No scratches or marks from previous use, and the fans are clean. Might never have been used. I wanted the x750e as it seemed to offer a good compromise of being cheap to buy, offered most (?) of the available features of pfSense and has the 8 network ports. I’m going to run it on a 4GB CF card for now (another cheap eBay purchase) but I can add a proper hard drive or SSD perhaps in the future. All good.
Having got the component parts together, I brought it in to work yesterday to ‘work on it during lunch’ 😉 I needed a CF card reader to blitz some images on to the CF cards but I knew someone would have one. First hurdle. Having borrowed it, I found my HP zBook 15 with Windows 8.1 wouldn’t see the reader properly. I have one at home somewhere but god knows where. So a colleague helped me out. I’d read that I needed a 256MB CF card initially to write the BIOS flash image to, and most of them come with a 256MB card initially. Mine didn’t, a bright, shiny 512MB card staring at me. No worries, borrowed a 256MB Cisco card off a colleague. He wrote the image to the card and it was time to get it on to the box. The aim of the BIOS flash is to enable BIOS is to enable console access to the BIOS so that a few tweaks can be made in there to have it see the larger card. Worth mentioning here that apparently, M0n0wall will install to the standard, much smaller CF card without any BIOS shenanigans, so if this makes you nervous, bear that in mind. I keep a USB to Serial cable in my bag in case it’s needed for datacentre work, I connect to the kit with internet access and share my desktop for the network guys to do their stuff. I tried this using COM4 (discovered by opening Device Manager and looking under ‘Ports (COM and LPT )’) and the recommended Putty settings. This brought up a console session but no flashing cursor. Several cables later from various people, I found one that was correct. This was a harder task than it should have been, there are lots of cables that have right connector, female to female 9-pin to 9-pin, but they can have the pins wired up differently. You can read lots about this and it got very technical. At last, someone handed me one that worked, a black Maplins item for the record, and now I could connect straight away. I knew the image on the card was good as I was getting the 3 beeps on boot, indicating it had booted to a prompt. I ran the BIOS flash to B6 and went in to set the BIOS settings as advised. This is where it went a bit wobbly. pfSense is a community based project and as such, information appears buried deep in forum posts and slightly convoluted ‘official’ documentation. I was told later on the forum that B6 is old now and 8.1 is the newer version, with useful modifications. I was also using the latest image, v2.2, and why not use the latest version 🙂 The combination of these two tripped me up. I left it yesterday afternoon at the point where I had inserted the 4GB card with the v2.2 image written to it, but it was not booting. I just had endless retries of ‘ata0: DMA limited to UDMA33, controller found non-ATA66 cable’. I would would retry 3 times, exhausting it’s retries and then starting to try again. I mentioned this on the forum and it’s a simple-ish fix, a few tweaks to a couple of config files and all should be good.
https://forum.pfsense.org/index.php?topic=20095.msg480824#msg480824 for the v2.2 booting situation, and :
https://sites.google.com/site/pfsensefirebox/home/XEBIOS_81.BIN for the later BIOS.
I’ll report back when I step through this and have it up and running.
All in all, a fun little project that should teach me a thing or two. Not as easy as I thought it would be, although the forum is very good with very fast (and patient) answers to my newbie questions. There a re a few further tweaks and tricks to get things like the LCD display and indicator lights working properly, something that an impatient chap like me would like to see baked into the image, but then this is free, and is the nature of community-based projects. It’s not beyond me, and will be all the more satisfying when I’m done.
If you’re thinking of attempting the same, have a good read around, there’s a lot of information about this out there, so much in fact that it can be tricky to find exactly what you’re looking for.
Start at : https://pfsense.org/ and register on the forum. You will have questions to ask 🙂
Another useful resource is this page : http://practicalkungfu.net/2012/02/20/how-to-install-pfsense-2-0-on-a-watchguard-x750e-core/ It might not actually work out as simply as this but it’s a great step-through of the required steps, and something I kept referring back to.
I’m sure I’m not alone in this thinking, and doubtless others have spoken about the same thing. IT is in itself a fast moving beast and we all know this. I’ve traditionally been what I consider Microsoft-centric. I sing the ‘everything is awesome’ song to myself each and every morning on the way to work. However, even just over the past 6 months perhaps, I’ve found myself irresistibly drawn towards new and exciting ways of doing things.
I find myself reading about Docker, and Chef, and other such technologies that seem really new (although they probably aren’t, I just haven’t been reading the right blogs). It’s interesting that building a VM in Azure now has the option to bake in an Octopus agent (hilariously termed a tentacle), and a Chef agent. Microsoft appears to be embracing these technologies more and more, HDInsight pulls in Hadoop, for example. DevOps seems to be all anyone talks about now, and I reckon to have a good spin on what it is, but what it means TO ME is that I should be getting more involved with the dev side, something that turns me white. It’s not that I don’t like the pasty faced hermits (lol), it’s just I’m not a coder and don’t walk in that world of thinking. Is it going to be as easy as people say? Do these two worlds comfortably combine in a happy, smiley way? Another change of thinking.
When Microsoft themselves are using third-party, and often Open Source technologies, it feels as though I’m going through a shift. Perhaps the days of being a Microsoft guy are fading, and it’s time to embrace a new way of thinking. The business has always termed our department ‘platform’, which I was never keen on but that might be a better description now than when it was thought up.
So the IT landscape is still a changing one, but it’s not only getting faster with accelerating Microsoft releases, the mindset required to succeed is evolving just as fast.
I got a new laptop at at work recently, an HP Z Book 15, which is a good machine. Very slick, and used the included SSD for the OS and got a caddy to replace the optical drive with another old-school spindle and platter hard drive for data, ergo VM’s. In my rush to get it sorted and be back in play for work stuff, I installed the same old VMWare Workstation 7 and upgrade to 8 application and license as I had on my old laptop. This didn’t want to play ball, so I had the bright idea of enabling the built-in Hyper-V which has worked OK but is overkill for simple labbing of ideas. VMW WS is nice and slick for quick and easy stuff on a laptop, it’s easy to template the various OS builds for cloning and sharing host folders into VM’s is a simple and painless procedure. Networking is also easier, I’ve found Hyper-V wants to use my wireless adapter for it’s networks and this has caused issues with wireless generally, instability and an unwillingness to connect just when I really need it. So I ended up with, you guessed it, lots of virtual network adapters spread around the place that seemed to be causing further issues. I decided to uninstall VMWare workstation, only to be faced with an error on the uninstall. The installer obviously performs a check on the host to see if Hyper-V is enabled (logically), and errors out very early on if it is. The thing is, I was trying to uninstall it but it just wouldn’t get that far. After some digging, there’s a way to ‘temporarily’ disable the Hyper-V hypervisor with a bcdedit tweak.
All this info is from https://communities.vmware.com/thread/305746 but the long and the short of it is that the information in the second post did indeed work like a charm. I tweaked, rebooted, and was able to uninstall VMWare WS without issue, and all the network adapters it creates for itself have gone. I revisited the command prompt and reversed the second command to set the hypervisor to ‘Auto’ (bcdedit /set {guid_hvoff} hypervisorlaunchtype auto) and rebooted again. I might change the boot file back to remove the choice on boot of HV or no HV, or I might leave it, perhaps it’ll come in useful at some point in the future. I didn’t see the same issues as the original poster in that thread, following this second reboot, I was able to open Hyper-V manager and start one of my VM’s, indicating that the hypervisor was happy and hadn’t been uninstalled. I point out I’m using Windows 8.1 and he was using Windows Server 2008 R2, so this is obviously the difference.
A strange place to find myself in. I’ll continue to use Hyper-V for now as it works and I can use my VM’s created in there for labbing and testing, but I have to say that I’m chasing a WS10 license to get that back on so I’ll uninstall the Hyper-v feature/role, install WS10 and convert the VM’s from Hyper-V format to WS format.
It’s entirely my fault for rushing things and getting in a self-made muddle, with 2 hypervisors present, but that’s how it is at work sometimes.
Taken from the Born To Learn website (http://borntolearn.mslearn.net/), this is a good deal. It’s really worth keeping an eye on this site for offers like this, for a long time I felt the site was a bit woolly and din’t pay it much heed. I do come back to it every now and again for a check though, as it’s worth a read. Offers like this can add up to a good saving over time.
There’s regularly news on exams, and I also see there’s talk of a new type of exam question : ‘Assertion-Reason’. It’s discussed in : http://borntolearn.mslearn.net/btl/b/weblog/archive/2014/10/27/introducing-a-new-question-type-assertion-reason-tell-us-what-you-think.aspx and the feedback…? Well, make up your own mind 🙂
Consider this a retweet or something similar, but it came up in my Old Reader RSS feed list, and I thought it quite pertinent to my general content.
Nice to Know–Deploying Windows Server Technical Preview on HP Proliant Microserver Gen 8
It’s actually another issue where the network card driver is really important, a colleague and I found an issue a while back where a server properly refused to talk properly to the network due to the driver applied to the network card on a new build. It wasn’t of our doing originally, we first got hold of the issue long after chins had been scratched and brows furrowed, but it was a strange one to undo, we didn’t for a moment think a new build could go out without correct and current firmware and drivers and it never crossed our minds to check until we’d exhausted most avenues….
I do tend to drive others mad with my obsession with drivers and firmware, but I find it an important part of really solid and proper server build. I can’t even run test VM’s on my Win8.1 client Hyper-V without having them patched up-to-date. Hardware and software manufacturers don’t release these things for fun, and granted, in a production environment, testing is key, if you’re lucky enough to have resources to do that, but these things should be kept up to date in a structured and managed manner 🙂
Just thought I’d post this again, it’s worth noting, and I can imagine anyone affected will have a painful day 😦
http://www.channelregister.co.uk/2014/04/25/hp_proliant_firmware_issue/
If you’re running HP servers and actually have a regular schedule to patch hardware (firmware and drivers), beware :
“On certain HP ProLiant servers, certain HP Broadcom-Based Network adapters listed in the scope may become non-functional when they are updated with the Comprehensive Configuration Management firmware Version 7.8.21 using firmware smart component, HP Smart Update Manager or the HP Service Pack for ProLiant 2014.2.0,” the firm stated.
Take care and stay classy 🙂
Was passed a query the other day where the login details for an iLO2 (HP DL360 G5, iLO FW – 1.61, 31/8/08) had been forgotten (……..). This sort of thing beefs me and it shouldn’t be difficult to keep a track of these things, but politics aside, the issue was there. The server had become unresponsive some days earlier and further confusion and disorganization led to a right old faff overnight.
So, how to reset? Well, coming to the server now, it’s up. Downtime is generally undesirable in a production environment, whether planned or planned, five 9’s and all that. I could simply bounce the server and F8 into the iLO configuration at boot, but this is downtime. It’s to be expected this method should be used at initial build and implementation time, but this was quite obviously a server that had been whirring away for years. HP do provide a GUI utility to provide this exact required functionality (http://h20566.www2.hp.com/portal/site/hpsc/template.PAGE/public/psi/swdDetails/?sp4ts.oid=1135772&spf_p.tpst=swdMain&spf_p.prp_swdMain=wsrp-navigationalState%3Didx%253D%257CswItem%253DMTX_e958118899b649d88441609f3f%257CswEnvOID%253D4064%257CitemLocale%253D%257CswLang%253D%257Cmode%253D%257Caction%253DdriverDocument&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken), and while on site the other night, I ran the installer which ran through without issue. You need to look in the ‘Installation Instructions’ to see that it does have a GUI part to it, it doesn’t seem to be mentioned elsewhere. I wasn’t aware the utility had previously been installed and was presented with a dialogue box asking whether I wanted to update to the latest version. Yes. It’s worth noting the majority of the utilities add a Start menu entry on installation, but this one hadn’t. It installs to <system drive>:\Program Files\HP\hponcfg, hponcfg.exe being the command-line part and hponcfg_gui.exe being the GUI-based version. Fantastic, let’s do it and get home. Apart from the fact that although the utility itself does not require a reboot, it does need ‘one of the following versions of the .NET Framework : v2.0.50727’. Ah, more downtime. As this was a LOB server, I stopped at this point. Always a good tip, work as far as you have agreed to and if thrown a curve ball like this, stop, go home, and rearrange stuff the next day.
I’m now left with the command line version of the utility… A quick Google reveals it works from importing .xml files, not my strong point. No-one seems to give much of an explanation as to the step through key-presses to sort this, so I spend around an hour going over the pdf that explains it all : http://h10032.www1.hp.com/ctg/Manual/c03351064.pdf, the iLO 2 Scripting and Command-Line Guide. A quality read for the beach, I can assure you. But it’s still not made clear enough for a dunce like me. I did then find this : http://community.spiceworks.com/how_to/show/2931-reset-hp-ilo-2-password-without-server-reboot. My appreciation to the writer, it made it straight-forward enough to do. Everything else points to changing the user login details once you’re logged in with something, but I couldn’t. I could get local access to the server’s console though so could run the utility. I did take away from the write-up that it’s a safer option, certainly in my situation, to create a new dummy user with adequate rights, log in with that and then do any required admin. This leaves you with some flexibility in case of problems. I created a temp user with admin rights, logged in, created the correct user, logged out, tested access with the correct user, and then with that login, deleted the temp user.
Something I didn’t find at all until this article is the HP Lights-Out XML Scripting Sample for Windows (Google and download it). This .zip file contains 72 files, most of which are .xml templates to perform various functions. This makes it infinitely easier for a non-coder like me to get things done. When the HPONCFG tool runs, it applies either an entire .xml file to the iLO, or a portion of a complete config. So I used ‘Add_User.xml’, opened it in Notepad, amended the various required fields, and wrote it with the utility. It’s funny, looking at the list of files now will make a Powershell ninja chuckle, most of these files are pre-fixed with GET, SET or MOD, it’s the verb-noun syntax 🙂 The GET_ files, for example, should be good for programmatically pulling back server hardware details, very interesting.
And there we have it, a little extra effort and a server reboot avoided. Takeaways?
Build servers properly in the first place (grrrr). Yes, use the PSP or Intelligent Provisioning as it is now. But check the driver and download page for the product in question, and check that these useful little utilities are installed and readily available. The major bonus of expensive enterprise-class server equipment is the supportability and hopefully up-time of the device. This can become limited when you realize you need a certain tool, and it hasn’t been installed, or won’t run without a reboot-requiring patch or whatever. I’ve heard of the scenario where, although unlikely and unrealistic, customers refuse to allow a server to be downed for trouble-shooting, and in this extreme, it’s too late.
Keep tools and firmware (and drivers) up to date. Create a schedule for planned maintenance and regularly check for updates to these things. Even if you’re a shop that doesn’t do these updates on the basis that if it ain’t broke, don’t fix it, at least document updates available so that if you see this or that problem, you have a fix to hand. A test environment certainly helps here, but good backups go a long way to reducing risk. It’s not uncommon to see considerable performance and stability increases from a simple driver update.
Test tools and diagnostic pages regularly to make sure they work. It’s usually when something’s on fire that you need a tool that worked two years ago but is not now. Regular OS patching, and absolutely critical and unavoidable process, might result in a tool ceasing to work, a later version will normally accommodate such changes.
The final note has to be documentation. The ultimate from this perspective would be a pro-active team who ensure everything is kept up-to-date as a scheduled and pro-active action. Regularly testing iLO logins etc. Ensuring that documentation is kept current is a huge battle, and can depend on how quickly your environment runs and changes. But it’s SO worth while trying, even if it’s a lever-arch file in a locked cupboard. You never know when you might need that obscure password. In a hurry. 🙂
homelabblogger 