Approaches to consider for new or existing system setup
Hi all,
I compiled some notes after helping a Vet to rescue his system and understanding some of the problems faced by Vets in choosing and running their computer systems.
This is not deeply technical but more about the important options you can consider in setting up and running your practice systems reliably with low cost and stress.
I have attached a word document hoping it may be useful to some.
Best Regards
Colin
Re: Approaches to consider for new or existing system setup
Some interesting points....I will say - there are a number of ways to correctly configure both vpns and allow a server running Openvpms to access the internet.
Openvpms server requires an internet connection to send mail to clients - send sms's and place orders with a supplier..isolating this system from the internet means your providing a hobbled service that the client cannot utilize to its fullest capability.
A good device based VPN/Firewall solution running Open/Tomcat on SSL and decent passwords for users provides at least 4 layers a hacker would need to brute before they got in - and setup a couple of systems to monitor security looking at attempts and you can react to them pretty quickly.
seriously if some russian hacker stole my data - I would just advise the police and run with my backup -- who is going to buy that data - I will tell you this - spammers or marketers looking for contact details - thats it...if one of my competitors was stupid enough to buy it - well I have enough clients who are bonded to find out pretty fast if they started getting reminders from another practice - and at that point...from both a legal and financial standpoint the other practice wouldn't exist very long - courts in Australia take a very dim view of corporate espionage - the fines and jail time are huge not to mention the punitive damages would render most small business owners bankrupt.
Re: Approaches to consider for new or existing system setup
T
The attacks are automated, they do not know or care who you are or what your data is until much later generally. They do not need passwords to get in they can get in via an email or a web page, THEN they will get the rest of your passwords. Alternatively, it is actually usually surprisingly easy to get enough passwords to get in. Even the most savvy will cop this one way or another eventually.
True, the data is not that interesting. However by the time that is established, your system will be quite sick and compromised.
It may well be possible to improve security for Vet systems. Generally speaking though, this will be VERY expensive for most of them and they will be even so taking some ones word for things.
For the Vet I was helping, it appeared to be a job yet to be done and he did not have the experience to diagnose or resolve issues that should not have been there been there from day 1. I strongly suspect that this is a job to be done for many if not most Vet system and if so many of them are at risk now.
The help he was paying a lot for, appeared not to be helping and appeared to be making things worse. This was VERY stressful. My recommendations will be very effective, at lowering stress, risk and cost, excepting physical onsite attacks, by staff say, and there is no charge.
The Vet should evaluate the alternatives paying attention to what might work against what will almost certainly work because there is no way of mounting an internet attack on a system that is not connected to the internet. Remember, these things are not generally targeted, they target every one detected, automatically. If it was targeted, you would not last a day but generally Vet systems would not attract targeted attention of course.
By the way, after I posted this, to the implementation subsection under the user’s forum, I moved this to the general discussion, because I do not regard it as a technical discussion which is how the implementation discussion within the user forum appear to be flagged and flavoured.
I strongly object to it being moved to an area where Vets "Users" generally will not look especially without a link, and especially without checking with me. Please move it back and leave the link to it from this implementation forum. This is a User forum, one would think it is FOR the Users, not for Techos to decide who should see what and where. I am representing a User here and it should be my choice who I am targeting one would think.
Best Regards
Colin
Re: Approaches to consider for new or existing system setup
It was appearing in both places - sorry I moved it to the one I thought most appropriate - given it address's implementation.
I agree with your points that most vets simply dont have the security or the setup that they should have - but there are many many good resources out there that deal in much more depth with the various issues surrounding server build and hosting that are far beyond the scope of a discussion on a forum aimed at the support of OpenVPMS.
I would ask though that if you feel there are any issues at all with security as provided by default on a Openvpms download that you bring them up here(by here I mean let us know so we can fix them or patch them yourself and send it over :) )...the source code is public ...so any vulnerabilites are also public.
Re: Approaches to consider for new or existing system setup
Thank Ben,
Yes and those resources, are generally for small business, expensive beyond belief and more often than is comfortable, absolutely not good resources. My intention is to put more power into the hands of the Vet to control his risk, cost and stress.
I will include a section on "doing you own recovery, to another system" sooner or later, as I would think that most Vets would have to hire help to do this, and this should not be necessary as they should be trained, preferably via written instruction, and practice this weekly on their second machine.
Security issues, by their nature, are never explained to the general public or in this case the forum, until they have been fixed, or the whole world will have a easy way in and plenty of time to practice. I doubt that I would have to tell most of the highly competent technical community what they are, but there is clearly a section, perhaps small, of the technical community, that have no idea, and are causing damage and expense and great stress.
My intention is to provide general advice for the Vet on how to remove the risks by using a general, easy and cheap, strategy. It is after all, for the Vet to choose and this is simple, easy to understand and run and very effective. The system proposed is working fine for the Vet whose system I helped rescue. He is very happy. And had he started out with this set up, I cannot imagine him having the problems he did, nor the expense.
The only problem he would have had was that the system was so badly tuned from new, as delivered, it had unbelievably bad performance such as 18 minutes to print a label generally and this once got out to nearly an hour believe it or not. The highly competent support service thought a new server would do the trick, at $8000 plus, but now, the same original system, "tuned up a bit" is now achieving 3 second max to print a label and pretty much instantaneous response at the terminals.
So we have two problems there, the person who originally built the system, and the person who was supposed to support it ongoing. Oh and some of the license material can not be found although everything provided was filed and is still filed.
Fixing the security issues individually, involves more time and cost than is really practical for a small business such as a Vet. The issues are generally not with the application, although there is one large entertaining gap in OpenVPMS itself that I am sure every OpenVPMS developer knows about.
I clearly think it is in the Vet practitioner community interest that they see this post and the attached document of course. Anyone who agrees, feel free to point people to it. Thanks and
Best Regards
Colin
Re: Approaches to consider for new or existing system setup
I am going to assume you are referring to
http://www.openvpms.org/project/enhance-openvpms-login-security
as the "entertaining" gap.
If not - can you provide more details please.
Re: Approaches to consider for new or existing system setup
Ben,
Can I get your number in the white pages, suburb??? If not do you wish to post a number, preferably a landline business number but up to you, and I will call with details.
Best Regards
Colin
Re: Approaches to consider for new or existing system setup
I think I should add that my recommendations are aimed to apply to the systems used to support most internal small business packages. I think OpenVPMS is a great option for Vets and the whole value proposition and community support is great and as far as I can see, probably unbeatable.
My advice is about setting up the system so that the package of your choice runs, run well with lowest cost, risk and stress.
To put it another way, you will face very much the same issues I cover in choosing and running the underpinning operating system as they are not related to a choice of OpenVPMS or any other package. They come with having to run a small business computer system of almost any flavour.
The exception to this may be if providing a Web presense from you business server. Most small businesses outsource web presense. Regardless, people rarely run their web server facility very close to their main internal business system as it required quite a different security setup and management.
Re: Approaches to consider for new or existing system setup
Colin - in the 1.8 version of the Context Sensitive Help we have added a How To section (see http://www.openvpms.org/documentation/csh/1.8/how-to ) which incudes an "Administering OpenVPMS" section. I would like to include some of your ideas. Can you please clarify a couple of points in your document:
a) backup - you say "The automatic backup system installed by OpenVPMS service integrators using snapshot is very good". Can you clarify the specific mechanism - are you referring to using mysqldump ? [see http://www.openvpms.org/documentation/csh/1.8/topics/how/backup ]
b) tuning - you say "I have seen an OpenVPMS system go from 18 minutes (yes minutes) to print a label to about 3 seconds or less." Can you provide some guidance on what you did to achieve this? My own experience is that tuning is black art, and I would love to have a simple 'do this, do that' set of rules.
Regards, Tim G
Re: Approaches to consider for new or existing system setup
Hi Tim,
Of course, happy to have my input used anywhere it can help.
a:
The backup system I am referring to is in principle great but the particular instance we have running on server 2003 R2 is of course now somewhat out of date.
I am not sure where it came from but it is in a folder called vcbackup and users Server 2003 RSM. RSM (Removable Storage Manager) is not around on Server 2008 and beyond so this script needs an update to work on server 2008 or 2012.
Still the sequence supplied on our 2003 system works like this;
Firstly about 11pm system timer event starts an SQL dump to export the mysql data base on a daily basis to an area on a system hard disk. The file get over written every day.
Secondly, about 2am so about 3 hours later, a second system timer event kicks off a VCbackup which takes a "Snapshot" (Snapshot product) image of the system disk and then another Snapshot image of the application disk is take and written to an RD1000 drive with file names that indicate date and time of the backup.
The RD100 cartridges are replaced in a cycle so if you have 5 then you put Mondays one in on Monday and enter the fact in a paper diary using a pen. (low tech but useful). Then on Tuesday you plug in the Tuesday one and not this in the diary. The diary is useful also if the responsibility to put the tape in is often changing.
Any cyclic system works, so you can use 7 tapes or 10 or 14 or whatever suites.
The result of this is that there is a system image taken every evening, which includes a fresh mysql backup, should you wish to use it.
We found this to be very useful, as our system was effectively destroyed by external parties allowed in through the router.
By the time we fixed the problem, all terminals except the system console were not working, so the business could still work but things were difficult as one can imagine with 5 terminals down.
We had to go back to get a working copy of the OS disk and the OPENVPMS application disk from a few months back, as in before the system had been hacked, and then restore the latest (as in todays) openvpms mysql image. This backup system allowed us to achieve both of these backup requirements.
For us, to move this backup "VCBACKUP" script to server 2008 or 2012, I need to get a fresh copy of the script that will work on 2008 and or 2012 and would like to ask if there is one available.
A couple more comments, it is of course useful to delete old backups so that the RD1000 do not fill up.
So with 5 disks, 160 gig each, they can take at least 10 15gig backups each, so across the 5 disks you can go back about 50 days and that can be VERY useful.
Also, I have realised that the RD1000 is potentially quite a unique device as removable hard disks go but I assume many people are now using NAS or SANs so potentially do not have much interest in RD1000s.
The RD1000 unit actually acts like a tape drive but using disks. So the system address as in C: or F: or H: is set when the RD1000 unit is cabled up to the mother board and gets a set GUID. The cartridges when plugged in are therefore just media. This appears not to be the case in any other removable disk technology, where a change of disk would change the GUID and even perhaps the disk address, as in H: could become J: if you were unlucky. Not the case with the RD1000. If anyone knows of another item that has this capability I am interested.
b:
Tuning. Yes a bit of a black art but usually quite easy once you are familiar. There are areas of speciality but generally and in approximately order of priority;
Of course, look at CPU load.
Except for very "big end" design mainframes which can schedule work to use 100% of the CPU without affecting high priority work, so we are generally speaking about most wintel (Windows on Intel) boxes.
Wintel boxes should be running probably not more than say between zero and say 70% CPU. So if you see 100% CPU all the time you almost certainly have a problem. Park that observation for a second while you look at the next item, then we will know where to look next.
So next;
Look at storage usage (memory, not disk). If you have a 32 bit system you will probably have only about 3gig of memory available to the OS and the Application before the system is forced to page heavily. So if you have 3gig and you are using 4 gig your system will be "thrashing" and performance will be beyond belief slow. If you can improve this system requiremnt to using say 2.5 gig, the system will be so much faster it will be difficult to believe the difference.
So frankly, first, make sure you are not using more memory than you have. In our case, we had VMware sitting there, not required, and mysql was using way more memory than we could afford. Getting rid of VMware and reducing the mysql usage from about 1.5gig to 384 meg fixed the memory problem.
Once that was fixed, we still have very high CPU, which was a five year old copy of Kaspersky. Got rid of Kaspersky and the system was responding pretty much instantly. Installed Microsoft Security Essentials, and still no problem. So basically, it is CPU and memory first. Get that sorted and if you still have problems you may have to look at disk but unlikely. SSDs are now a big help if you really need high speed disk, but it IS NOT the solution to a high paging rate. Either get more memory or use less memory.
Same for CPU, if you have done your best and are still out of CPU (not likely these days because CPUs are now so powerful) either get rid of some load (in our case the old Kaspersky was using about 80%) or get a faster CPU or system.
But then for us and not so relevant to tuning;
Due to a router setup issue, that we were not fully aware of at first, we were getting increasing instances of external parties into the system. We quickly found the router issue but by then we had people in generating bitcoins, checking their Facebook pages (still being slack here on sending their pages to the AFP) and 3740 viruses of various sorts on the system. Bugs can really slow your system up. That is partly why I suggest keeping your business system off the internet. The risk is way higher than most Vets realise. I will not even offer to prove this to any one as I don't feel like causing trouble for myself or anyone else but rest assured that most vets at least, are highly vulnerable if they leave their business systems attached to the internet and their backup system had better be working which means testing a recovery about once a week on another system is my preference.
Hope this helps. We could build on this to flesh out details if required. These are the main tuning areas. If you are short on CPU and or memory and the requirement for CPU or memory cannot be fixed or lowered, get more memory or CPU, nothing else comes close and tweaking is pretty much a waste of time and money. I am happy to have a look at any systems people believe are not working well and provide a few suggestions or explanations of what is going on.
Best Regards
Colin
Re: Approaches to consider for new or existing system setup
Colin - thanks for your input. Your backup procedure logically matches what we do. Your tuning notes match what we went through. I would agree that memory allocation in a Win-2003 environment is problematical - and we plan to move to Win2012 next year. You didn't mention the java memory allocation - which can be important in a large system.
I am not sure of the quality of the current M/S Security Essentials - in other support work I have found that it is not nearly as good as say Avast or AVG (for protecting naive/stupid users who do not use 'safe surfing' practices). We do run MSE on the practices' servers but we do have a premium router (SonicWall) out the front.
We also cannot take the 'disconnect from the net' approach. Apart from losing OpenVPMS's email and SMS facilities, we could not run the housecall business, and remote support would not be possible (the practice is in Hong Kong, I live in Oz) - as well as the loss of gmail, google calendar (used to schedule the housecall vet), the staff staff scheduler, etc.
Regards, Tim G
Re: Approaches to consider for new or existing system setup
Hi Tim,
yes the tuning is not a mystery to the IT savvy but to others, they are exposed to advice along the lines of "You need a new bigger more expensive system" for example, so basic guidance and actual unbiased guidance can be critical. Always seek a few views on your performance issues might be a good approach.
MS also have an advanced version of MS Essential, I think it is called Forefront or similar. Not sure about pricing, if any but it is packaged somewhere in the server 2012 material and I got the impression you can just unpack it and install and there may be no license issues perhaps if you are using it on Server 2012. I did not have time to investigate further.
As for connecting to the internet, of course each to his own. However, in reality pretty much no large organizations run their business systems behing one fire wall, there are always two. and the only systems that are exposed behind the first firewall, are web server and email servers. All othe internal systems are behind the second firewall. This allows the internal systems to talk to the web servers and email servers but generally not the internet. Bit over the top for an OpenVPMS system. I would quite like to demonstrate the weakness there are in play but they are sure to cause considerable annoyance and people have been dragged off by the AFP for dabbling in these areas so I am staying well clear.
By the way, I have been inside some of the largest IT companies on the planet while thery were being invaded because some one opened an email or browsed to a bad web page, so no amount of money of know how will protect you if you are connected. The G20 attendies are in the news now for the same sort of problem and they are behind their governments security systems.
Also remote maintenance access can be turned on or connected when required. If my Vet wants help he phones me and then attaches his system. After his experience, he would rather leave his car in the carpark with the keys in it than leave his system connected. Also, he has no problem using an email system that is not connected to his server but may have less functionality than you prefer.
I hear your views, but I would suggest that the openVPMS teams have an ongoing aim to improve functionalily AND isolation, so how can I get this functionality AND the safety of distance from the internet. Bit of ingenuity needed perhaps, and maybe a few compromises, but this is a goal that I magine you would be happy to incorperate as a guiding principle if you do not already have it as one.
Also, can any one share a server 2008 and a server 2012 script for doing the Snapshot daily backups suitable for use with and RD1000 please??
Thanks and
Best Regards
Colin