How do you know who is accessing your server?

In case of TCP protocol i.e. ServerSocket:

Each Socket connection accepted corresponds to who is connecting to your server, that means relevant method calls on ServerSocket will fetch IP address and port of the same.

Socket socket = serverSocket.accept();

// Print IP address and  port
System.out.println ("Connecting from : " +
 socket.getInetAddress().getHostAddress() + ':' + socket.getPort()); 

In case of UDP i.e. DatagramSocket

The DatagramPacket received contains all the necessary information:

DatagramPacket datagramPacket = null;

// Receive next packet
datagramSocket.receive ( datagramPacket );

// Print address + port
System.out.println ("Packet received from : " +
 datagramPacket.getAddress().getHostAddress() + ':' + datagramPacket.getPort());