Unable to Create Objects/Add Hosts in the Sun MC Hierarchy

[Bobby Saini]

The goal of this forum entry is to assist Sun Management Center (Sun MC) users with one of the most common problems, which is adding monitored hosts to the Sun MC hierarchy.

When this fails, you will see the following error message in the java console, specifically the Create Object window:

Creating node...Please Wait.
About 1-2 minutes later you will see...

Cannot find agent process. Please check host/port.

This error message can occur for several reasons. These reasons and their associated debugging solutions are summarized below. Before moving on, I would like to point out a few things about the "Create Object" window:

a) Usually, when trying to create an object, it will only take a few seconds. If 20-30 seconds pass by and the object is not yet created, it most likely will timeout and fail with the above message.

b) Only the Hostname or IP Address field needs to be filled in, not both. No harm in filling in both, but its usually more successful to have the IP Address and filling in the both will cause the server to use the Hostname instead of the IP Address.


Steps to Determine the Cause and Solutions to this Problem:


01. Is the Sun MC agent running?

Open a telnet or SSH session to the problematic agent host and run the following command to verify whether or the not the agent is running:

# ps -ef | grep esd | grep agent

If it's running you will see the following:
    root 28142 14380   1   Oct 16 ?          17:19 esd - init agent -dir /var/opt/SUNWsymon -q

If it's not running, please start it up:
# /opt/SUNWsymon/sbin/es-start -a

02. Verify the hostname or IP Address and the Port number for the agent are correct. Also make sure this agent is in the correct server context (reporting to the expected Sun MC server)

# cat /var/opt/SUNWsymon/cfg/domain-config.x

Look for the agent section and verify the hostname and port number are the ones you are using in the "Create Object" window. If you like, you can change the agent hostname to use the IP Address.

i.e. In the examples below, "britney" is the agent and "paris" is the server.

agent = {
        agentServer = "britney"
        snmpPort = "1161"
        logicalAddressMode = "ip"
        historyOutput = "clog://localhost/../log/history.log;lines=12000;width=200;flags=rw+;mode=644;mmap=false"
}

Anytime the hostname or IP Address or Port numbers are changed, you must make sure to reseed the agent. This is because SNMPv2u communication is used for authentication and files need to be updated with the new changes. To reseed, run the following command:

# /opt/SUNWsymon/sbin/es-run base-usm-seed.sh -s <SEED> -c agent

where <SEED> is the seed value provided on the Sun MC server during the setup portion of the installation.

Note: If the <SEED> values are not the same, you will have this very problem.

Lastly, make sure you are actually logging into the correct Sun MC server (If you have more than one). Sometimes users login to their 2nd Sun MC server, but the agent is configured to communicate with the 1st Sun MC server.

You can verify the Sun MC server context by looking at the trap and event sections of domain-config.x

trap = {
        trapServer = "paris"
        snmpPort = "162"
        logicalAddressMode = "ip"
}
event = {
        eventServer = "paris"
        snmpPort = "163"
        logicalAddressMode = "ip"
}

03. Can the server communicate with the agent and vice-versa? Command line test...

This step will verify whether or not any firewalls exist between the server and agent. Run the following SNMP commands on both the server and agents and look for any kind of output. No output indicates a problem with the communication:

On the Sun MC Server host (Using the example paris server and britney agent):

# /opt/SUNWsymon/util/bin/snmpwalk -h <AGENT> -p <PORT> 1.3.6.1.4.1

where <AGENT> is the AGENT IP or Hostname and <PORT> is the agent port number.

i.e.
# /opt/SUNWsymon/util/bin/snmpwalk -h britney -p 1161 1.3.6.1.4.1

A whole bunch of data will be returned indicating no communication problems. If the above command fails, you will see:

Request Timed Out: /opt/SUNWsymon/util/bin/snmpwalk -h britney -p 1161 1.3.6.1.4.1

You should also run this command on the agent (britney) to verify it can communicate with the Sun MC server (paris).

# /opt/SUNWsymon/util/bin/snmpwalk -h paris -p 1161 1.3.6.1.4.1

Assuming the agent port is 1161 (Can be verified in domain-config.x).


04. I've done all the above and still cannot successfully add the agent. What should I do next?

The following steps will be useful in debugging the problem:

On the agent system (britney)

a). Stop the agent

# /opt/SUNWsymon/sbin/es-stop -a

b) Change your domain-config.x file on the agent to use IP Addresses instead of hostnames:

# vi /var/opt/SUNWsymon/cfg/domain-config.x

Change all entries of hostnames to IP Address.
i.e. Change britney to its IP, change paris to its IP.

c) Reseed the agent

# /opt/SUNWsymon/sbin/es-run base-usm-seed.sh -s <SEED> -c agent

Make sure <SEED> is your seed value used when the Sun MC server was setup. If this is incorrect, then you will get the behavour you are now seeing.

d) Restart the agent

# /opt/SUNWsymon/sbin/es-start -a

e) Watch the agent.log file for the esmaster entry:

# /opt/SUNWsymon/util/bin/ctail -f /var/opt/SUNWsymon/log/agent.log | grep esmaster

Once you see the esmaster entry in the log file, this means the agent has been authenticated by the Sun MC server and you should then be able to add the object.

f) Use the Java console to add the newly monitored object to your domain using Edit -> Create Object.

For troubleshooting you can also try running the snoop command on both the server and agent to watch network traffic between these two systems:

# snoop udp between <SERVER HOST> and <AGENT HOST>
ie.
# snoop udp between paris and britney

As you start the agent, watch the snoop output to see if the appropriate communication is happening.

If all the above still fails, please take a look at our forum entry:
http://forums.halcyoninc.com/showthread.php?t=2

Sometimes this problem occurs due to firewall or multiple network interfaces.

[Pegah Garousi]

Adding a node for a Windows Agent is another common problem faced by Sun Management Center users.

You must first add an icon for the Windows Agent in the SunMC Console.
Using the Sun Management Center console, do the following:

1. In the main console window, select "Edit" from the menu bar and then "Create Object...".
2. Select the "Node" tab in the resulting dialog box.
3. Select "SNMP Ping" from the "Monitor Via" picklist.
4. Enter a label in the "Node Label" field, e.g. britney (XP)
5. Enter a description in the "Description" field, e.g. Windows XP Host on 4th floor, main building
6. Enter the hostname in the "Hostname" field, e.g. britney
7. Optionally enter the IP address in the "IP Address" field, e.g. 192.1.4.5
8. Enter the port number in the "Port" field. This is the port number you provided during installation . The default is 161.
Note: If it was necessary to install the PrimeAlert Agent for Windows on another port (e.g. because there was already an agent using UDP port 161) then specify the port you used.
9. Click "OK". A new node is inserted into the topology window.
10. In the main console window, select (click on) the PrimeAlert agent host that was just added to the current domain.
11. Select "Edit" from the menu bar and then "Modify...".
12. Select the "Node" tab in the resulting dialog box.
13. Select "Sun Management Center Agent - Host" from the "Monitor Via" picklist.
14. Click "OK".

If after doing this step you are still having problems then follow these steps:

Run the snmpwalk command from your Sun MC server to the agent:

This step will verify whether or not any firewalls exist between the server and agent. Run the following SNMP commands on both the server and agents and look for any kind of output. No output indicates a problem with the communication:

On the Sun MC Server host (Using the example paris server and britney agent):

# /opt/SUNWsymon/util/bin/snmpwalk -h <AGENT> -p <PORT> 1.3.6.1.4.1

where <AGENT> is the AGENT IP or Hostname and <PORT> is the agent port number.  If you do not have the default snmpv1 community of "public", then also put "-c <COMMUNITY>" before the 1.3.6... oid

i.e.
# /opt/SUNWsymon/util/bin/snmpwalk -h britney -p 1161 -c public 1.3.6.1.4.1

A whole bunch of data will be returned indicating no communication problems. If the above command fails, you will see:

Request Timed Out: /opt/SUNWsymon/util/bin/snmpwalk -h britney -p 1161 1.3.6.1.4.1

You should also run this command on the Windows agent (britney) to verify it can communicate with the Sun MC server (paris).

Open a command prompt and do the following:

# cd C:\Program Files\Halcyon\PrimeAlert\install\bin\
# snmpwalk.exe -h paris -p 1161 1.3.6.1.4.1

Assuming the agent port is 1161 (Can be verified in domain-config.x).


Try modifying your Windows "domain-config.x" file and reseed the agent:

Go to

C:\Program Files\Halcyon\PrimeAlert\local\cfg

Open the "domain-config.x" file.

If the trapServer and eventServer are specified by the Server Host name, change them to the IP Address of the Sun MC server.

While we are here, also change the port number of the agent to 11161.

Now reseed the agent:

Run a snoop on your Sun MC server so we can watch the traffic going in and out:

# snoop udp between <Sun MC server> and <Windows Agent>

In command prompt, run the following:

01. Stop the Halcyon PrimeAlert Agent service

# net stop "Halcyon PrimeAlert Agent"

02. Reseed the agent:

Note that when viewing our README_INSTALL for the Windows Agent via a web browser, a few parameters don't show up in the reseed command due to us forgetting to escape "<" characters. The -p parameter is also misleading, it's not what people expect it to be. The -p parameter should not be used, you should be using the -u parameter instead if you need to specify a non-default snmpv1 community. If you have previously used the -p parameter, specify it again with an argument of public.

# cd C:\Program Files\Halcyon\PrimeAlert\local\cfg
# ..\..\install\sbin\paAgent - tool usm-seed -s <master_seed> -u <snmpv1_community> [ -p public ] agent

i.e.
# C:\Program Files\Halcyon\PrimeAlert\local\cfg>..\..\install\sbin\paAgent - tool usm-seed -s mySeed -u myv1community -p public agent
WARNING ... agent.agentServer not found - using britney

Ignore the Warning.

03. Restart the agent:

# net start "Halcyon PrimeAlert Agent"

You should see traffic in your snoop output such as:

$WINDOWSAGENT -> $SERVERHOST         UDP D=162 S=$AGENTPORT LEN=164
$WINDOWSAGENT -> $SERVERHOST         UDP D=163 S=$AGENTPORT LEN=207
        $SERVERHOST -> $WINDOWSAGENT UDP D=$AGENTPORT S=163 LEN=63
$WINDOWSAGENT -> $SERVERHOST         UDP D=162 S=$AGENTPORT LEN=198
$WINDOWSAGENT -> $SERVERHOST         UDP D=163 S=$AGENTPORT LEN=80
        $SERVERHOST -> $WINDOWSAGENTIP UDP D=$AGENTPORT S=163 LEN=102
$WINDOWSAGENT -> $SERVERHOST         UDP D=163 S=$AGENTPORT LEN=1421
$WINDOWSAGENT -> $SERVERHOST         UDP D=162 S=$AGENTPORT LEN=202

Where:
$WINDOWSAGENT: is the Host/IP of your Windows Agent
$SERVERHOST: is the Host name of your Sun MC Server
$AGENTPORT: is the Port number of your Windows Agent

i.e.

 192.1.4.5 -> sunmcserver         UDP D=162 S=11161 LEN=164
 192.1.4.5 -> sunmcserver         UDP D=163 S=11161 LEN=207
        sunmcserver -> 192.1.4.5  UDP D=11161 S=163 LEN=63
 192.1.4.5 -> sunmcserver         UDP D=162 S=11161 LEN=198
 192.1.4.5 -> sunmcserver         UDP D=163 S=11161 LEN=80
        sunmcserver -> 192.1.4.5  UDP D=11161 S=163 LEN=102
 192.1.4.5 -> sunmcserver         UDP D=163 S=11161 LEN=1421
 192.1.4.5 -> sunmcserver         UDP D=162 S=11161 LEN=202

04. Now try modifying the object:

a. In the main console window, select (click on) the PrimeAlert Windows agent host.
b. Select "Edit" from the menu bar and then "Modify...".
c. Select the "Node" tab in the resulting dialog box.
d. Select "Sun Management Center Agent - Host" from the "Monitor Via" picklist.
e. Change Port to 11161.
f. Click "OK".

[Craig Wyllie]

If the snmpwalk from the agent to the server fails, but your firewall admins insist they've got things configured correctly - it's possible they are only allowing traffic originating from the agent port. snmpwalk of course uses a random high numbered port.

See the last two steps of the following forum post, on checking commmunications using the MIB-II and MIB-II Proxy modules inside agents themselves.

[Craig Wyllie]

A few more things to verify

Verify that the user that you are logging in as on the Sun MC server
layer has the appropriate permissions to create a topolgoy object.

Verify that hostname translation to IP address and vice versa
on both hosts is correct for both hostnames.

Verify that you can ping the Sun MC server from the Agent host, and
vice versa, using both the hostnames and IP addresses.

Run ccat on the agentStatus.log and ensure that the IP Address being
used in the traps is the correct one for the Sun MC server. (It is
easiest to copy the ccat binary from it's installed location to the
log directory so you do not have to specify it's path.)

Run "ccat event.log | grep -i cold" on the Sun MC Server and ensure
that the coldstart trap from the agent is being received by the Sun MC
server and that it has the correct IP Address for the agent host.
If you don't see any matching lines "ctail -f event.log" and stop
and re-start the agent, watch for a coldstart trap that may have the
incorrect hostname/ip-address.

The snmp walk from the agent host should also be done against the trap, event, and security daemons, ports 162, 163, and 165 respectively. For example on a windows agent host:

. . . .C:\Program Files\Halcyon\install\bin\snmpwalk.exe -h SERVERHOST -p 162 -c SNMPV1COMMUNITY 1.3.6
. . . .C:\Program Files\Halcyon\install\bin\snmpwalk.exe -h SERVERHOST -p 163 -c SNMPV1COMMUNITY 1.3.6
. . . .C:\Program Files\Halcyon\install\bin\snmpwalk.exe -h SERVERHOST -p 165 -c SNMPV1COMMUNITY 1.3.6

If the Halcyon Agent is on Windows, this may include "personal firewalls"
running on the Windows host itself or Microsoft's "TCP/IP Filtering"
in the Network Configuration ("Advanced TCP/IP Settings" for the
Adapter/LAN, final "Options" tab).

Does the Halcyon Agent Host or the Sun MC Server Host have more than one NIC?

If so, does the Halcyon Agent hostname resolve to the IP address that is
associated with the NIC and network that the Sun MC server layer
can be reached through?

Again under windows - is the Windows nic in the top of the list of
configured NICs in the Windows network configuration? (Right click
"My Network Places" and select "Properties", then in the "Advanced"
menu select "Advanced Settings", within the "Adapters and Bindings"
tab ensure that the itmes that lead to the Sun MC server are at the
top of the list.)

If the Sun MC Server has more than one NIC consult the following Sun documentation:

. . . .Sun MC (ver) Installation and Configuration Guide
. . . ."Configuring Server and Agent on Multi-IP Machines"
. . . .http://docs.sun.com/app/docs/doc/816-2678/6m90st2ee?q=multiple+&a=view

Is the right hostname being used as per resolution through the correct NIC?

Bobby and/or Pegah had earlier suggested specifying ip addresses in the domain-config.x file on the agent side. Please note that for certain versions of the Halcyon PrimeAlert agent, you must add the "agentServer = IPADDRESS" line yourself, put it inside the "agent = {" section.

[Craig Wyllie]

Here's another simple case to check:

http://forums.halcyoninc.com/showpos...27&postcount=8

and a slightly more complicated procedure to try is described here:

http://forums.halcyoninc.com/showpos...24&postcount=6