Level: Introductory Eric Boehm (boehm@nortelnetworks.com)Nortel Networks
16 Feb 2005 This article covers compiling, installing, configuring and modifying Samba for use in a IBM Rational ClearCase interop environment, and includes some in-depth troubleshooting of common errors. This article describes a configuration that is based on standard templates and is used at most Nortel Networks sites around the world.
Background
This article derives from the following session of the 2003 Rational User Conference in Orlando, Florida:
SCM04 -- Setting Up and Troubleshooting an IBM® Rational® ClearCase®-Samba Interop Environment
Monday, Aug 25, 2:45 -- 3:45
Eric Boehm, ClearCase Performance Prime, Nortel Networks
Abstract
This article discusses compiling, installing, configuring, and modifying Samba for use in an IBM® Rational® ClearCase® interoperability environment. It includes some in-depth troubleshooting of common errors. This article describes a configuration based on standard templates used at most Nortel Networks sites around the world.
Related Presentation Files
The attached zip file contains sample templates, detailed instructions, and other resources for the procedures described below. To view the instructions, download the zip file to a local directory, extract the files, and then open the file samba.html.
Templates and detailed instructions for procedures
Setting Up and Troubleshooting an IBM Rational ClearCase-Samba Interop Environment
This article covers the following subjects:
- What is an interoperability environment?
- How to build Samba from the source code
- How to configure Samba
- Upgrading from Samba 2.0.X to Samba 2.X
- Troubleshooting Samba and Rational ClearCase problems
I would like to give you a little background about why I care to expend this much energy on the subject of setting up and configuring Samba. I started with Samba back in March of 1999 and had several servers to configure. It was a pain to have to configure each one by hand. Furthermore, we have a great variety in environment sizes: from 8 VOB servers with 500 VOBs to one VOB server with only 22 VOBs. In addition, we're spread out quite a bit; there are several U.S. and Canadian locations, along with international locations in Brazil, England, and France. Consequently, I worked hard trying to make the configuration and setup processes much easier.
ClearCase Interoperability Environment
In order to understand the information in this article better, it will be helpful to answer the following questions:
- What is a ClearCase interoperability environment? ClearCase supports a mixed environment of UNIX™ and Microsoft® Windows® clients.
- Why do we need it? You have to use an NFS or SMB server product to get the data to the clients.
- Why do we want to use Samba?
- It's Open Source, which means it's free.
- The configuration is server-centric -- I only have to configure it in a few places (I don't have to go out and touch all the clients).
I have tried TAS, Hummingbird Maestro, and other NFS client products, but I've had the least trouble and the best results with Samba.
Building Samba
Getting the source code is rather easy. The current version is 2.2.8a, which is approved for use with ClearCase. You can get the source code from the Samba website. Note that it is to your advantage to compile the source code as a 64-bit application. I had a problem in 2001 where I couldn't do a build. Even though I restricted mnode settings, it still wouldn't work. I came up with a patch to let me have more files open, and I submitted that to the Samba development team. 64-Bit compiling allows you to have more open files without having to restrict the number of mnodes (discussed later). Some operating systems might not need to be compiled as 64-bit. Figure 1 following addresses building Samba in a Solaris™ environment.
| |
Figure 1: Building Samba on Solaris
|
If you want more details about the patch, I have provided the following links:
In a 32-bit environment, there is another limitation involving the use of the system call, fopen. Samba uses this system call to open some files, usually log files. The Samba server can get into a state where there are more than 256 files open. Then, it will fail if it tries to open a log file or some other file. In this case, you can actually end up with a broken installation where Samba becomes inoperable.
Figure 2 shows how you might compile Samba as a 64-bit application with different compilers -- the Forte compiler is the one you have to purchase from Sun. You can also build this with GCC (GCC 3.x stream is required for 64-bit compilation).
| |
Figure 2: Building Samba with Sun Forte or GCC
|
You can also build this on HP-UX (Figure 3), and HP-UX 11.x is a 64-bit operating system. In the second bullet point shown in Figure 3, furthermore, there are some settings you might want to change in order to improve performance.
| |
Figure 3: Building Samba on HP-UX
|
An additional compile step concerns two utilities (shown in Figure 4) that are useful when you're running Samba. By default, the regular compile does not build these utilities, which help you manage the database of passwords and other keys that are important to the operation of Samba.
| |
Figure 4: Installing additional Samba utilities
|
Figure 5 illustrates some compile problems that you might encounter if you use the SWAT application, which is the web-based way of managing Samba.
| |
Figure 5: Potential compile problems
|
The observed problem behavior is that you enter your username and password to access SWAT, but it returns a blank page and nothing happens. It could be that SWAT is returning an SEGV signal. This happens because of a bad header, but this should be fixed if you're building the recent release -- they corrected that problem. If you see errors when trying to run SWAT, run:
CPPFLAGS=' -D_EXTENSIONS_'; export CPPFLAGS
The most common reason that a GCC 64-bit compile fails is somebody trying to use GCC 2.x instead of 3.x; as discussed previously, you must use GCC 3.x or later. Finally, there was an intermediate bug in the 2.2 stream: if you tried to build a shared library, the shared library would fail because it didn't pass all the correct flags to the compiler.Thework-around for that is just to specify a new definition for the SHLD macro:
make SHLD=' ${CC} ${CFLAGS} ' |
On HP-UX 11, if you use the option to build libsmbclient, it seems to fail, but this is not necessary for normal operations (it is a utility library you can use for your own programs). I haven't been able to determine why it fails or how to fix it, so the current work-around is simply to not include it. Finally, there are some additional options for the Sun compiler for building 64-bit (Figure 6):
| |
Figure 6: HP-UX compile problems and Sun Forte options
|
Configuring Samba
Concerning the configuration file structure, I made some changes to make it easier to deploy this to many servers without having to do much work on each server. By default, the configuration file is called smb.conf. I still have that file, but I have added two hooks to configure smb.conf.global.<hostname> and smb.conf.shares.<hostname>. The global file maintains parameters that are specific to a system. You only need to configure it when you initially set up the server; you don't need to go back and touch this file.
The smb.conf.shares.<hostname> is the file that configures the shares that are available from your Samba server. This is the configuration file that you'll be modifying over time, because that's where shares will be created, deleted or modified.
Example templates of these files are attached -- they are in machine-readable form. The initial file, smb.conf, shouldn't require any changes. The only thing you might have to do if you use a 32-bit operating system is to set max open files down to a lower value. However, I recommend that you not use a 32-bit operating system if at all possible.
You might have to make another change to the smb.conf file if you are going to run separate smbd and nmbd daemons on a system that has multiple network interfaces. We will discuss why you might want to have separate daemons later. If you are using multiple daemons, you want to keep your log files separate. Note that log files are very important because they can really help you debug problems. The default configuration I use has a separate log file for each client: if you only have one log file with a hundred or a thousand clients, it's going to be difficult to find the information you need in that file. The downside to this approach is that log files can be, by default, about ten megabytes. If you had a thousand clients and each had a ten-megabyte log file, that could use up a lot of space. Typically, it doesn't get that bad, but it's just something to keep in mind.
If you are using multiple daemons, then, replace the include directive %h parameter in smb.conf.global.<hostname> with the primary hostname. This allows you to have a separate file for each hostname/interface pair.
Moving to the next file, smb.conf.global.<hostname>, the first step you need to think about is how you're going to handle authentication. The authentication directives are: security, workgroup and password server. There are four options for the security directive, but I'm not going to go into detail here about what they all mean. If you want a more detailed explanation about the options, there's a file available from the Samba website called DOMAIN_MEMBER.txt that explains them. The preferred option is security=domain, which provides the best performance. The main point is that when security=domain, the Samba server is a full participant in the domain. It has a machine account, can pass authentication requests to the domain controller, and can let the domain controller handle authentication.
Moving along, the password server directive is the name of the domain controller against which you'll be authenticating. Things have changed somewhat in Nortel: previously we had separate resource domains for machines and users. You had to be able to specify both sets of domain controllers, because the Samba server must authenticate against the machine domain, while the users must authenticate against the user domain. I don't know if other groups still do this, but we are moving to a single domain for machines and users, and we're also moving to Active Directory.
You need the WINS server for a multi-subnet environment, as well as for browsing. You don't really have to support browsing for your Samba server, but the password server directive takes an IP address. It is important to note that although you can provide multiple IP addresses -- it won't break anything -- it will never use anything but the first one.
Configuring multiple network interfaces
Why would you want to do this? One reason is to manage performance by having a separate pair for each interface. The ClearCase Administrator's Guide talks about managing different ClearCase regions on different subnets, which implies different network interfaces. You may want to restrict Samba to a subset of the available interfaces in the server. In our case, we have a separate interface that is used for backups, so there's no reason to run Samba on the backup interface -- nobody's going to be accessing it from that location.
Examples
Following are some examples showing how you might modify the configuration files. For instance, you might want to run Samba on some, but not all, of the interfaces. In this example, I chose two private addresses and use the interfaces directives to list the interfaces you want to access. You need a second directive, bind interfaces only=yes, to complete the configuration. Samba listens on all interfaces and you must therefore specify bind interfaces only=yes to restrict it to the interfaces listed in the interfaces directive.
The previous example just had the smbd/nmbd daemons on some, not all, of the interfaces. If you are using multiple daemons, one (and only one), of the smb.conf.global.<hostname> files must include the loopback interface in the interfaces directive. Use the smb.conf.global.<hostname> file that corresponds to the primary hostname. The other smb.conf.global.<hostname> files will need a netbios name directive set to the hostname of the alternate interface. This is made clearer in Figure 7, where we have the host alpha, which is the primary hostname, and we include the loopback interface in the interfaces directive. The hostname beta is on an alternate interface, and is not the primary hostname, so we must include the netbios name directive for beta.
|
Figure 7: Configuring smb.conf.global.<hostname>
|
If you have security=domain, have multiple interfaces, and you're running multiple daemons, you need to have a machine account for each interface. You can't have just one for the machine, because each daemon is separate and needs to have its own machine account. The problem is that the command to join the domain doesn't allow you to specify an alternate configuration file. I provided some examples here of how you might work around this problem by renaming files and using symbolic links. Once you join the domain, you don't need to maintain that file by the hostname, because the actual running of the daemons lets you specify an alternate configuration file.
Configuring the Shares
Moving on to how you can configure the shares, smb.conf.shares.<hostname> contains a template for a single share. Most of the parameters don't need to be modified, since they're set to default or recommended values. The only things to be concerned about are the share name and the path to the shared directory. Following are some specific restrictions:
- The share name has to conform to NetBIOS naming restrictions, which means you can only use letters and underscore in the share names, up to 14 characters.
- There is a bug if you are still using Windows NT -- it might not work if the share name is more than 11 characters.
Keep these things in mind, then, depending on your environment.
Figure 8 lists a description of the various other directives that are common to a share. They're pretty self-explanatory. Use the hosts allow and valid users directives if you want to restrict access to a share to a certain group of machines or users. I've found in practice that the hosts allow doesn't work so well, especially if you're in a DHCP environment or have remote users whose hostname might change. It's better to go with valid users, because at least that way, if the user moves around to different machines, the user will still be able to access the data.
|
Figure 8: Configuring the smb.conf.shares.<hostname> file
|
Username and Password Concerns
Now I'm going to digress for a moment. One of the assumptions I made here was that you are somewhat familiar with setting up an interop environment. It's very important to have usernames that are spelled the same on Windows and on UNIX. Another important consideration here is the lmhosts file. It's how Samba can resolve NetBIOS names. Normally, the WINS server would take care of this for you, but we've had the experience that the WINS server infrastructure is not very robust or reliable. In fact, we have experienced frequent outages because the WINS server was having some problem. One way to work around this is to have an lmhosts file on the Samba server. The lmhosts file typically lists the domain controllers that you're interested in. That's probably safe to do because domain controllers aren't usually going to change hostnames or IP addresses very often. Changing domain controller hostnames or IP addresses will cause disruption in the Windows environment. It may be desirable to create lmhosts files on the clients to avoid similar problems (due to outages in the WINS server infrastructure); I give the path to the lmhosts files on the Windows clients (discussed further in the troubleshooting section).
The username.map file is a way to map an account on the Windows side to a different name on the UNIX side. The primary use for this is to map the ClearCase ALBD service account on the Windows side to a valid account on the UNIX side. The default ALBD user is clearcase_albd, which is a long name that you wouldn't normally see in a UNIX environment. In the past, names longer than 8 characters would cause a problem. On most modern systems, it probably doesn't cause a problem any more, but it's something to keep in mind. A possible solution to dealing with this clearcase_albd account is to map it to the VOB owner account, because you know the VOB owner is going to have access to VOB objects and VOB elements.
Another alternative is to create an ALBD service account that has the same name as the VOB owner account, but I don't recommend this because there are security implications. If somebody were able to log on to Windows as that VOB owner account, then they could go into the VOB and do everything that a VOB owner could do, which is probably a bad thing.
The last alternative is the one that we use: create an ALBD service account with a user name that's constructed from the VOB owner account, but you attach an index to it. For example, if the VOB owner account is vobadm, you might use vobadm01. If someone logs in as vobadm01, they couldn't do anything to the VOB because it's not the VOB owner name.
The thing to keep in mind about our solution, however, is that this mapping for Samba only takes care of accessing cleartext and source containers, which is what the ALBD service does for you. It doesn't take care of normal ClearCase access control. If someone were to log in as vobadm01 and try to check out a file, it wouldn't map their account to vobadm. It would still be vobadm01, which would probably be nobody as far as the VOB was concerned.
Many organizations are moving towards large, consolidated environments, as we are. We take several groups that used to be on their own servers and consolidate them into a single group of servers. In this case, there will be multiple VOB owner accounts, probably multiple design environments, and multiple projects. It can be a little difficult to come up with proper mapping in this context. Keep in mind that you can map multiple accounts to a single UNIX account, which is straightforward.
You cannot, though, map one ALBD service to multiple VOB owner accounts. In the following example where I try to map albd1 to three different accounts, it's only going to map to the last one listed. In the example below (Figure 9), for instance, I tried to map ALBD account albd1 to vobadmin, ccadmin, and vobadmcc. However, the albd1 account will only be mapped to vobadmincc (the last definition). Users may experience problems accessing VOBs owned by vobadmin and ccadmin, since albd1 is mapped to vobadmcc.
| |
Figure 9: You cannot map multiple VOB owner accounts to a single ALBD service account
|
Some additional considerations concern the ALBD service account and password updates. In our environment, corporate security requires all accounts passwords to expire every six months. We could not get an exemption. Of course, it's a problem if your ALBD service account doesn't work -- your users are down. Therefore, we had to find a way of dealing with this policy. If you can get away with the ALBD account never expiring, that's great. If you can't, then you have to be aware that if the ALBD account password expires or is locked out, you're going to have a lot of users who are down.
As stated, we are faced with the restriction that the ALBD password has to expire. We use ALBD service accounts with indexes to keep track of the ALBD service account. When it is time for the password to expire, we have a second account ready with a password that's going to expire six months hence. That way, we can have the users switch over to the new ALBD service account while the old one is still valid. The additional benefit of this is that if we can get people to switch over, then when the other account expires and fails it will affect fewer users, minimizing the impact. Then you can tell the users who are affected, "OK, you need to update your ALBD service account." The account that expired will eventually become locked after a few failures when people who didn't update try to log in.
If you're using all these accounts with indexes, remember to pre-configure the username.map file. It's important to realize that if you switch accounts, you must also update Samba as well. If you forget to update the username.map file, then you will have access problems. It doesn't matter which method you choose -- you can either keep incrementing the number, or you could just bounce back and forth between two indexes.
Upgrading from Samba 2.0.x to Samba 2.2.x
Samba 2.0.7 was the previous stable version, and now the 2.2 stream has been released. There have been several performance problems with the 2.2 stream, up until about 2.2.7. One issue that's particular to Solaris, even with the current release, is that you need some patches to avoid high CPU utilization problems. This is because they changed how they managed locks, from using shared memory to using their own text database. The text database uses the fcntl system call and F_SETLK to manage the locks. Performance can degrade exponentially as the number of locks increases. I haven't seen this in practice, because I never used Samba 2.2.2, but I've seen people on the mailing list who have had this problem with CPU utilization going very high. Once you get above 75 clients, this can become an issue. All these versions of Solaris have a patch available to address this problem, so I would recommend applying those patches.
Which version of Samba would I use? For everything other than Solaris, use the latest stable release, which as of this writing is 2.2.8a. If you can apply those patches shown in Figure 10, then again continue with 2.2.8. (There were some security fixes that were necessary to address some potential exploits.) If for whatever reason you can't use those patches, then 2.0.10a is an option.
| |
Figure 10: Samba patches for Solaris
|
There are some differences in the configuration of 2.2.8a, and some files that were individual files are merged into a new file. These two files are the machine account password file (named something like <domain name>.<hostname>.mac) and the machine.SID file (which contains a Security IDentifier). The attached slides of my presentation, SCM04.pdf, give a recipe for how you might go about installing a new version of Samba, creating a new directory structure, and copying over your old configuration files. The private directory contains the machine account password and the machine SID file. You might need to modify a configuration file if you were using the shared mem size directive. This directive was removed in 2.2, so you should comment it out. It won't hurt anything if you don't, you will just get an error message in your logs.
The other thing you may want to do -- because the logging behavior for 2.2.x has been changed-is to set the debug level to 1 to give yourself more information. If you don't set it to 1, the only thing you will see in your log is error messages. Sometimes when you're tracking down a problem, you want to see informational messages about connections and how a user tried to connect. You can, of course, set this level after the error, but then you've lost an opportunity to get information. You have to wait until the error occurs again, or have the user try again. It will just be easier if you have the information there in the first place.
When you're ready to switch over to Samba, perform the following steps:
- Stop Samba.
- Rename the directories.
- Do one final update of the private directory. (As I said, this machine account password updates weekly, and you don't know when it might have updated. Even if you've copied the files over ten minutes ago, it could be possible that it updated in the ten minutes that elapsed. You just want to make sure you're as up-to-date as possible.)
- Restart Samba.
It's less than five minutes to switch over, and it's pretty easy to back out.
Once it's done, you want to verify that it seems to be working okay. You could do that by going into a client and mounting VOBs. An easier way, however, is to use the smbclient command (which is part of the Samba distribution) to mount a share on the UNIX machine. Alternatively, on a Windows machine, just type net use on the command line. If the connection fails, then you have to go back and look at the log files to see what might have gone wrong.
We saw three main problems during the upgrade. The first one was a log message indicating that Samba tried to change the trust account password and failed. The problem is that when the 2.0.7 configuration is migrated to 2.2.x, it doesn't create a key in the text database that's needed in order to change the password. The Samba server will continue to work; it doesn't hinder performance. There is a potential security implication, since your machine account password is not changing, but it's not that severe an issue. One way to work around this is to have someone delete and recreate the machine account, and then rejoin the server. This may be disruptive, however, because all authentication is going to fail as soon as you delete the account, and your users could be interrupted. Then, of course, you would have to get the machine account created and rejoin the Samba server to the domain. What I did instead was to have a test server where I created a secrets.tdb file that had that key. There was other data in the file, but that gets overwritten when the server starts and migrates the data. That seemed to be the easiest method.
The second problem we saw was during an upgrade from 2.2.5. You start up the server and you can't connect. When you look in the log files, you see a message that it could not fetch the trust account password. The problem is that the key where it stores that information is malformed. This is a binary file, so you cannot edit and fix it. The easiest thing you can do is to stop Samba. You can either dump the information out of the secrets.tdb file, or copy the data over from the old installation to create a <domain>.<hostname>.mac file. On the CD, I also provide a little utility to make dumping the secrets.tdb file a little easier. Once you have re-created the <domain>.<hostname>.mac file, the Samba server will merge that into secrets.tdb and create the key correctly. This problem will likely only occur for a very small set of people.
The last problem was more difficult. After the upgrade, Samba was not resolving symbolic links in the UNIX file system correctly. Unfortunately, we only saw this problem on one machine and I wasn't able to reproduce the error. I had test machines with the same patch levels and same kind of configuration, and had no problem with them. You might see an error message about "not a valid win32 application" or "unable to run exec format error". In this example, we tried to access the file syee_test1.bat because the error messages in the log files showed that it left out part of the path. This would be a rare occurrence, but it's something to keep in mind when you're upgrading. You might want to check that symbolic links are working properly after an upgrade.
Troubleshooting Samba and Rational ClearCase
The log files are your friends; you can get a lot of information out of them, so obviously you need to find out how to increase the debugging level. The debug level is set to a value between 0 and 10. Zero is almost no detail, and 10 has a lot of detail. There are two ways to change the level. One is to edit the smb.conf file to change the directive, and then stop and start Samba. The other is to send a HUP signal to the Samba daemon. If you send a HUP signal, only new connections will get the new debugging level. I don't usually use this method, because it affects all Samba clients. When you're debugging, you usually want to debug only a particular client. To debug a single client, I go into the log and find the process ID for the particular client.
The way I have this configuration file set up, the templates have the process ID included in the log file. These configuration files have evolved over time, and there have been probably seven or eight iterations. I added information and changed options to make it easier to debug any sort of problems that might arise. In this example (Figure 11), the PID is 4539 and -- for the 2.0 stream -- you send a signal to the process. You can only increase the debug level by one each time a signal is sent. If you want to increase it 2 or 3 levels, you have to send 2 or 3 signals. For 2.2.x, they included a utility that lets you just set it to any number, and that makes things a bit easier.
| |
Figure 11: Debugging Samba
|
Figure 12 shows the error that I see most often. I have seen several postings to the ClearCase user's group for this problem. This probably is the most common error that you're going to see: you'll get an error message that reads, "Unable to construct cleartext for object X in VOB Error: Type manager 'text_file_delta' failed construct_version operation." This is almost always due to some sort of permission problem. What is interesting is how many ways you can have a permission problem. The other thing that the user might see is the error message "incorrect function", which will be in the view log and the view host. There might also be a pop-up dialog with this information in it.
| |
Figure 12: The most common error
|
How do we go about tracking down where exactly the problem is? The first thing you need to know is the ALBD service account name. You can find that through the services control panel. Then you have to ask yourself, does this service account have a corresponding UNIX account? If it does, you can go to the next step. If it doesn't, then it's probably mapped, so you want to look at the username.map file. If it's not there, then you have to add it. After you add it, new connections will pick up the new value, but existing connections won't. To solve this, the client that's having a problem will have to unmount the VOBs, wait a few minutes for the Samba connection to be deleted, and then remount the VOBs. It may be even easier to have the user reboot. Alternatively, you could stop and start Samba, but that's going to affect everybody instead of just the users who are having the problem.
Does the UNIX account have access? This assumes that the ALBD service has a UNIX account or is mapped properly. Does that account have access to the VOB storage directory and subdirectories? I see this a lot, where the UNIX permissions under the VOB storage directory are something like 700 or 750, which makes it hard for anybody but a small group of people to access the data in the VOB. The solution to this is simple -- just correct the permissions on the VOB storage directory. If there are problems within the pools themselves, you have to use the cleartool protectvob command to address that. You don't want to mess with those permissions with UNIX commands.
The other thing that could happen is if for some reason you have to have restrictive group permissions. You must make sure that the UNIX account used by the ALBD service is in the right group, and if you encounter a problem you can either correct the permissions or add the user to the correct group.
Difficulties Mounting VOBs or Accessing Shares
Even after going through all of the above, you might still have trouble accessing VOB data. There are lots of other ways that things can go wrong. Typically, what you have to do next is increase debugging levels to get more details about what's happening. The first steps I usually take are to try to get to the shared directory without going through ClearCase, to make sure that that part of the networking is working OK. I use a net use command and try to mount the share. This is where you would hope to see a failure. If the user specified the wrong password, you can look in the Samba log, and the message is pretty clear. It says: "NT_STATUS_WRONG_PASSWORD". This is likely to occur if the user has just changed the password, or changed the password recently without loggin out and back in again. The solution is to do that -- log out and log in.
The user account could also be locked out if the user changed the password and tried logging in with the incorrect password too many times. The error message in the log is again fairly self-explanatory, that the account is locked out. You just need to wait a few minutes for the account to be released, or if that doesn't work you have to ask the user to have their password reset.
A harder-to-debug way that access can be restricted is if you use the valid users or the hosts allow directives that I mentioned before, in which case only certain users or hosts can connect. If you see a prompt for a password after the net use command, and you enter the correct password but the command still fails, then you have to look at the shares configuration file and see if there are restrictions on the share. If there are share restrictions, you will need to investigate whether the user or machine should have access. You must increase the debugging level to about level 5 in order to see the error.
In the example shown in Figure 13, the smbd daemon is looking for my userid, boehm, in the SUPPORT_USERS group. My userid is not found in the group, so I am not allowed to access the share.
| |
Figure 13: Troubleshooting VOB mount and share access
|
The other thing that can happen is -- whether intentionally or unintentionally -- that the user's account gets deleted or doesn't exist. Then when I try to access a share, they will come in as user "nobody" without access to the share, and the operation will fail. You really need the log to see that error indicated. (Figure 14), boehm-1 connects to service ccaserls as user nobody. Instead of user boehm, it says user nobody. That's in the case that a UNIX account is missing.
| |
Figure 14: User's account does not exist or has been deleted
|
We ran into still other problems with the WINS server. Users could mount the VOBs and they would be working fine; then, suddenly, they couldn't access or mount the VOBs. What we found was that the WINS server might go belly up, or we might get tombstone entries for our VOB servers in the WINS server database. That would be bad, and the work-around we found for this, which is not great, was to create lmhosts files with entries for the VOB servers on the clients. This helps speed up access to the VOB servers, because the client doesn't have to wait for WINS resolution to find the VOB server. In addition, VOB servers shouldn't be changing hostnames or IP addresses very often, if at all. The disadvantage is that it can be a lot of work if you have to create this file on every client, and you would have to make sure that new clients get this update as well. We have addressed this problem by putting a wrapper around the installation process for ClearCase. It creates this file with the correct entries during installation.
The file that you need to create is in the %SystemRoot%\SYSTEM32 directory. There's a sample file in that directory, lmhosts.sam. In Figure 15, I show an lmhosts file based on the servers I used earlier; the #PRE tells it to pre-load the hostnames in the cache, and it makes look-ups go a little faster.
| |
Figure 15: An example of the lmhosts file
|
Yet another way that you can fail to mount the shares occurs if the Samba server machine account is corrupt. This can be indicated if when you do the net use command, you're prompted for a password and type the correct one in, but it still doesn't work. You will see error messages in the logs like the "NT_STATUS_NO_TRUST_SAM_ACCOUNT" or "NT_STATUS_ACCESS_DENIED". This is a little more disruptive because this affects all the users, and not just a single client. Typically you won't have any permission on the domain controller to fix the machine account, so one work-around for this is to go to 'security=server' until you can get the account fixed. Once it is fixed, you can pick a more opportune time to rejoin the domain and restart Samba. This process would result in a 2-3 minute outage.
The last errors are ones that you should not see any more. The "Out of file structures" error was very common when trying to open too many files with 32-bit Samba. The error is indicated when you try to read a dynamic file and you get an "incorrect function". You shouldn't see this any more, provided you're using a 64-bit Samba and version 2.2.x.
One note here is that Rational's documentation still states to set mnodes to 800. I've been using higher values for mnodes since January of 2001 without any problem.
The solution to the problem is to increase 'max open files' and use a 64-bit Samba. If you're using a 32-bit Samba and increase the value of 'max open files', it won't help. If you can't use 64-bit Samba, then you have to decrease the mnode settings. These are in the ClearCase control panel under MVFS performance. You could start with 800. You may even have to decrease it as low as 200. If you still have a problem, which is what happened in my case when I originally developed this patch, then you simply must get to a 64-bit version of Samba.
The last problem is one that's rarer still, or should be becoming more so. Samba 2.0 manages locks with a shared memory table. It's easy with 100 to 200 clients to exceed the size of this table. The default size is about a megabyte. In the configuration files I provided (attached), I set this value to about four megabytes. Even so, once you start getting 100 clients or more, it's very easy to exceed that four-megabyte level, and then you'll see messages such as "panic error, delete share mode hash bucket empty". It is particularly frustrating, because what the user will experience is that they could read this file previously, and now they can't. They might reboot their machine or restart their view. Now they can read this file, but they can't read a different file that they could read before. It all depends on just how fast the shared memory table filled up, and what file they were trying to access at the time that it was full.
The solution is to increase the shared mem size directive. Again, this is for 2.0: it shouldn't be a problem for 2.2. If you increase this parameter, you probably have to increase the kernel parameter as well. The kernel parameter for Solaris is shmsys:shminfo_shmmax. HP-UX has a similar setting. For a change to a kernel parameter to take effect, you have to reboot the system, which is, again, even more disruptive.
If for some reason you had the kernel parameter set high enough that you could increase it, one thing you have to know is that if you change the size, you have to delete the existing shared memory. This is because the Samba code knows to go and look to see if the memory table is out there, and if it finds the shared memory it will try to reuse it at the wrong size. Figure 16 shows the commands to remove the shared memory and the semaphore. The keys are fixed; they're hard-coded into the Samba source.
| |
Figure 16: Increase the shared mem size directive
|
Questions and Answers
What do you do with Samba if you can't go to a 64-bit Samba?
The solution is to go to the Rational ClearCase control panel. There's a tab called MVFS Performance with two check boxes (maximum number of mnodes to keep in the VOB free list and maximum number of mnodes to keep for the cleartext free list). You can manually override those and set them to 800, and you might even need to set them as low as 200.
Your examples for Samba configuration were Windows NT-centric. How much does the information change for an Active Directory environment?
The question was about PDC (primary domain controller) and BDC (backup domain c controller) in an Active Directory environment.
I don't know the changes you need to make -- I mean, you still have domain controller in Active Directory, so you still need to have the passwd server, which would be whatever your active directory servers are. I don't know if WINS server is still used in Active Directory. I believe it still is, because I think we're using it. So you still do need that information even in an active directory environment.
What directory does Samba keep its logs in?
If it's installed in /usr/local/samba, the log files will be in/usr/local/samba/var. The log will called log.<machine name>.
How do we deal with the expire password for multiple ALBD accounts?
We have our own tools for updating clients, but perhaps the easiest way for general installation would be to go into the release area and update the sitedefs.dat file, which contains the encrypted password. You would then run siteprep.exe again, providing a new password. It will update that file, and then users would essentially re-install ClearCase to update that password.
Does a 64-bit Samba require a 64-bit operating system?
Yes it does. You have to be running Solaris 7 (64-bit) or better, or HP-UX 11.
Do you have a problem if you're running Linix in 64-bit?
No, because Linux doesn't suffer from this problem. There are different settings you might have to tweak, but on Linux you don't have this problem with the fopen system call, and there are some kernel tunables that you could set for Linux that would allow you to open enough files.
Problem with caching -- change is made on the server but clients don't see if for a few minutes.
The question was that someone is seeing a problem with caching where they make a change in the Solaris server, but the client doesn't see the change until a few minutes later. What that might be is in my configuration file there's an option called op locks, and op locks=Off for ClearCase use, because that's what it does. It caches things, and it can cause problems. So I don't know what you have yours set to, but that would be my first guess concerning what might fix it.
If you have users logging into multiple windows domains, do multiple daemons take care of that problem?
That could be one way to address it. We have multiple daemons in our environment, but we didn't need to address it because there were trust relationships among the various domains. The problem you might run into may be that someone in one domain tries to log in, and they may not be able to authenticate if they got the wrong domain controller (where their account didn't exist, even though there is a trust relationship). One solution for you might be to have one interface have a machine account in one domain, and another interface have a machine account in another domain.
How do I handle UNIX account synchronization?
It hasn't been a problem because all the UNIX accounts are handled through NIS. If the UNIX account changes, it is handled by NIS. The authentication really doesn't happen on the UNIX side. If I come in as user boehm and connect to the Samba server, it just looks to see if a user boehm exists on the UNIX side. If I exist, fine, then it passes the authentication request to the domain controller; it never involves the UNIX side.
Has anyone seen a problem with Samba where the member usage increases and they have to stop and restart Samba?
I haven't seen that. I've seen discussions of it. I know that there were problems with the earlier 2.2 versions. I believe that they have been fixed as of 2.2.8a. If you're still having those problems, I don't know what to tell you. It could be an operating system-related patch, but I do know that there were memory leaks in the earlier 2.2 series.
Have you seen better results with Samba than TAS?
Yes. I took a look at TAS. I guess I didn't like the licensing cost. We had other sites within our organization that were using TAS, and one site started having problems. They couldn't authenticate users correctly because there was no access. We contacted Support, but we couldn't get any satisfaction out of them. A thousand users were down and couldn't do their work. Fortunately, we were able to switch over to Samba in a few minutes. I spent time creating the configuration files. We switched them over, and they were happy. We had a similar instance again in Brazil just recently where TAS was just not behaving properly, was doing weird things. Again, we couldn't get results from Support, so we switched them over to Samba and things were great.
Another thing that makes Samba a little better from my perspective is this: valid users and hosts allow. I can use NIS netgroups there to determine who can access my server. On TAS you have to edit a file listing all the user IDs on each TAS server manually. In RTP, I have 8 servers. I don't want to have to edit eight files on servers and keep them all up to date and in sync when I could have one NIS netgroup that has the data and is managed that way.
Do you have problems with having PC NFS on a client and determining which protocol it's going to use to access the VOB server?
No, the ClearCase documentation is rather specific. I believe it tells you how to address that. We have had that in the past and what you do is go into the services control panel, go into the properties and there's a service provider order. It will have something like let's say "Microsoft Windows Network", "Hummingbird Network", if you're using Hummingbird Maestro, and then it will specify some other option. You can change the order of those protocols so that you could put "Microsoft Windows Network" ahead of the NFS protocol, and it will use Samba instead of NFS. I had to do something similar when trying to debug a problem in the past: I wanted to force it to use NFS, so I changed the order and put NFS ahead of the Microsoft Windows Network.
What's the capacity of the Samba server?
The biggest one we have is one I mentioned in the beginning. We have a single machine in Canada that services probably a thousand users. Probably 600 or more are Samba or PCs or Samba users. It's an Enterprise 6500 with, I believe, 8 processors, and I've never had a problem with Samba where it took up too much of the machine to support users. I tend to have multiple VOB servers because I like that configuration better for performance, so I usually have several hundred users spread out around 2 or 4 servers, and I've never had a performance issue or a capacity issue with Samba using this configuration.
Do I put Samba in the VOB server?
Yes, that's where it goes.
About the author  | | | Biography to be updated. |
Rate this page
| 
