Starting to think extreme frugality on the part of management will cost me my job

len69 · 5 months

Be truthful and state your case to your bosses.

If they don't give you the budget you need to guarantee a minimum amount of security and backup procedures, there is just so much that can be expected of a sysadmin.

Sometimes it takes a disaster for non techs to understand the importance of spending the correct amount on building in safety measures.

SEA-Sysadmin · 5 months

Get out. Fast. Nothing will be more detrimental to your resume than having been fired. This company will never pay you a decent salary, give you a raise, or offer you an opportunity. You have already reaped any benefit of working there. Stop thinking about this and asking people- you know exactly what you have to do.

whateverradar · 5 months

Let me give you the best piece of advice:

Stop bringing them problems and start bringing them solutions.

give them options:

For X cost I can have us back up in x time For y cost I can have us back up in Y time.

I would prefer x, even though it costs more because it will prevent situations like this from happening again.

iamadogforreal · 5 months

Sorry, but I worked in a similar environment for a short while and chose to get a new job. If you make the case for RAID (hardware or software) and they won't go for it, then they're too rotten to fix. A server without RAID isn't a server, its a desktop pretending to be a server.

Just the idea of putting a used disk in a server sends shivers up my spine. Disks are constantly failing. The fail rate for a disk jumps up pretty high at the 3 year mark and then gets pretty bad every year going forward. I find most of my fails are early on (within 6 months) or at the year 3 mark. I can't imagine putting in a 2 or 3 year old disk into a server without RAID, or even with RAID! You bosses should really understand that if this was a major fail, say multiple disks all going out, they've more or less cost themselves 10x the cost and time of preventative maintenance.

phrstbrn · 5 months

It's highly unlikely you're going to change your boss's mind. Basically, there are two outcomes:

1) Your boss's boss will take the heat, be the fall guy, and then you might see some changes. This only works if your boss isn't the head honcho, and your boss isn't chummy with the head honcho

2) You'll be the fall guy

It doesn't sound like your boss is taking you seriously, so either he's going to fail or you're going to fail. You probably know better than I which scenario is the more likely outcome.

It's possible things will turn for the better for you, but if it's been more than a year and they don't take you seriously yet, it's likely not going to change anytime soon, if ever.

All that said and done, do you really want to be there when shit hits the fan hard enough that they have to make some changes? Of course assuming you don't take the fall for it first.

NatasReficul · 5 months

Ya gotta speak they language. Throw down some figures for down time, lost data, man hours, etc. Ask them, how much their data is worth to them? With 45 employees, a couple hours of downtime can be a sobering figure.

beedogs · 5 months

ugh, get out ASAP.

g-me · 5 months

Very similar situtation to me.... Even today I am rebuilding from a failed drive and restoring.

Try and get users to nudge management. Say to them that when X was down for X hours Y amount of workers couldnt work.

get the users frustration onto managmenet to make them more aware of the issues

TheGraycat · 5 months

Have you done a full DR or risk review for them? Basically break down all the risks, what they are, where they are and what the cost to the business would be if something happened. Make sure you spell out down time in cost per hour to the company (not just your time but lost time, project delays, other staff sitting around doing nothing etc etc). Then outline some solutions and their costs.

Once you've given that to them it's a business decision and not yours any more so you're off the hook.

In the mean time you can start looking at low cost / free solutions yourself :)

sylver_dragon · 5 months

Definitely feel your pain. One man shops have a bad tendency to rely heavily on bubblegum and bailing wire to hold everything together. At my last job, I felt such a sense of relief and joy as I turned off my last Win2k box for the last time (In 2008). This box had been a used HP server bought on EBay and was originally running damn near everything.

As for "solving" your issue. If you have a few spare hard-drives hanging about and enough parts to boot something, you might consider putting together a very basic backup server, toss a bunch of disks in it and setup a software RAID. Grab an OpenSource backup solution (Here is one which jumped to the top in Google). And just get something for the business critical stuff. Given the lack of resources, you are probably going to have to slice and dice the data very thin. Just get the stuff to allow bare minimum operation. Your users may have to supply you this information; to which they will reply, "it's all critical!" You'll just have to work them down to the basics.

Best of luck to you, and keep telling yourself "this is a great learning experience." Once you hit the 30-odd hour mark, you should be too tired to realize it's bullshit and it'll make you feel better.

castufari · 5 months

Document, document, document. When I have problems like this I email the director and financial people and let them know what has happened and what the solutions are. I end it with a "let's meet about this asap", then follow it up with an email, one that I send to myself. I try and use return rcpts.

In my old job we had a server that was failing. The box was 7 years old when I took the job, they wouldn't buy a new server nor would they buy new hard drives. They would - in 2001 - buy flat panel monitors for the "police" because they had an application that "wouldn't run on CRTs" (the vendor said it didn't matter, the "chief" had the sales rep put this on the quote and told the guy that if he didn't mention the monitors, they'd get no sale). The tape drive failed so I was manually copying everything to another box at night. A few weeks before I left the server bought the farm and they were trying to fire me. I told them that they'd be welcome to but, before they flamed me to staff, to look at my emails and notes about what was failing and what needed to be replaced. I got a new server out of it. :\ Then I quit.

Also, work on your own documentation - your resume.

xenu99 · 5 months

You need to speak in terms they understand, dollars and time, not technical.

$500 drive vs 45 employees.

$500drive/45employees =11.11 each. Assuming they are on $12/hr, the down time for 1 hour would equal the cost of the drive.

How much down time is going to be caused by the lost backups? How much business is that going to cost them. Heck, tape drives are dirt cheap now days. A decent back up system with offsite storage will cost under $10k/yr, which is probably a pittance compare to what lost data will cost them, that is, unless of course, they want the data to be lost for, um, accounting reasons.....

dtfinch · 5 months

They're gambling the company to save $50 on a hard drive, $500 on a backup server.

Though it's good to document these things (management has very bad memory when the time comes to point blame), it doesn't help much if the company goes under anyway.

At an early low point at my job, I had servers backing up the most important data to each other, and spare backups on my own desktop. No redundancy. No air conditioning. One domain controller. It got better though. Fear set in, and now everything's on raid 1, with multiple backups, onsite and offsite, keeping multiple past revisions. I don't have identical spares of our Windows servers for test recovery though.

spif · 5 months

I hope you've been putting all this in writing, including requests for resources to do backups, and detailed explanations of why you need it and what the potential consequences of not having it are.

The most important thing most admins forget when dealing with limitations imposed by management is to GET IT IN WRITING. Sometimes even that isn't enough, but it's better to do it than not.

Hachya · 5 months

For those that are in our IRC chan: And this isn't even ME!!

Step 1 DOCUMENT ALL THINGS!

Now, furthermore:

rant/

Eh, I have had to deal with this too many times, as IRC knows.

I had to buy used shit, 5 year old machines, for my monitoring boxes...

And, even after that, they wanted me to then cannibalize one of those and put it into a production environment, thus ruining my plans for one of them.

I ended up having to purchase a $99 rackmount server (see unixsurplus.com). Worst part is, again, it's a live DB that runs the heart and soul of one of the companies, their online sales DB for the store.

My new SQL and TS servers are underpowered CybertronPCs which I haven't even bothered to set up yet, time being a factor. I get those and also need to get rails, and am having a bit of a problem with that. I'm like, yeah you kind of need rails. They've even asked me to use an old Optiplex tower with 512 RAM at one point. (I said "fuck you")

My desktop is one of the oldest ones here. The newer ones they got they gave to people who only need to run Remote Desktop Connection, Mail, and Word, all the while I sit here with a 5 year old machine, grinding away when I'm running, well, anything.

It goes on and on and on.

All I can say is that I have had success when planning things out a year in advance and mentioning them with such a window. If you cam tell them that you absolutely NEED something and give probable cause displaying such a thing, it can help. (I'm not saying it will work all the time.) Another thing you may notice is money being spent in other areas that don't need it. DO NOT GET INVOLVED IN THIS POLITICAL BULLSHIT.

Explain it like this "if I don't get this, this goes down, and this and this" "And if THAT goes down we make nothing" or, "you cannot email clients, period"

I have made the mistake of buying/getting underpowered shit and have paid the price. Find out the bare minimal specs, and then make sure you got a couple levels up from there, you will thank yourself later.

Your liver will also thank you.

Also explain that without proper redundancy or another backup machine available, it may be a week (or whatever time you think it may take) to recover everything. And, when you say this make sure they understand that means THERE'S LITERALLY NO FUCKING REASON TO COME IN TO THE OFFICE.

And, if they even START to point to the cloud, they deserve a slow painful death.

/rant

diachi · 5 months

Sound exactly like my work ... number of offices and everything. As suggested elsewhere and be truthful with management. Other employees might get your back too - in my work the lack of money spent on infrastructure drives employees up the wall.

molandsprings · 5 months

One of the biggest struggles IT people have, of any size department or company, is to communicate in business terms the value of IT expenditures. If you do this correctly you have no blame. Far too often, however, IT people are either silent or aggressive. Neither works well. Not saying this is you, but it's food for thought.

atlanticblue · 5 months

As long as you get paid well I don't see the problem. If you have alerted them to the problems and they don't want to spend any money, 'be it on their own head'. Do what you can to save the data, but this isn't your fault if management declined the use of the required equipment.

For those that are in our IRC chan: And this isn't even ME!!

you'll need to login or register to do that

create a new account

login