Changeset 4114


Ignore:
Timestamp:
2010-07-05 07:57:19 (3 years ago)
Author:
bruno
Message:

Lily server process:

  • Add an indexer kauri module and make it listen to the repository queue.
  • Make hardcoded blobfs path (hdfs://locahost:9000) configurable.
  • Some fiddling with dependencies to make everything work in Kauri:
    • exclude Avro dependency from HBase, otherwise it is in Kauri's shared classloader, and it won't find the repository implementation classes. Seems like this needs to be improved in Avro: it should use the ContextClassLoader? for loading classes instead of Class.forName.
    • commons-httpclient needs commons-coded, but there were different version of commons-codec in use, causing it to be in the module-private classloaders, while commons-httpclient was being shared

Remote communication: use the HTTP connector for Avro, instead of the simple socket one. With the socketserver, I noticed that after the first avro request, the cpu of my laptop went to 190% and stayed like this until the server process was stopped. With upcoming avro 1.4 there will be a Netty-based connector that we could use.

Added a simple import tool to load field types, record types and records from a json file. The trigger to write this was to be able to test the standalone Lily without writing code. It is not yet feature complete (e.g. mixins not supported for record types, no possibility to update records) and there are no tests yet. Thanks to the power of Jackson (the json parser, not the singer) it was easy to do the parsing of the json in a combination of streaming and tree parsing, allowing any size of input json file to be processed.

Misc:

  • Remove non-existing artifacts from root pom.
  • Add missing AvroFieldTypeNotFoundException? for getFieldTypeByName
  • Fixed switched repo and typemgr hosts in Client.
Location:
projects/lily/trunk
Files:
30 added
19 edited

Legend:

Unmodified
Added
Removed

  • projects/lily/trunk/pom.xml

    r4111 r4114  
    6565    <module>process/server</module> 
    6666    <module>process/client</module> 
     67    <module>import</module> 
    6768  </modules> 
    6869 
    … …  
    107108      <dependency> 
    108109        <groupId>org.lilycms</groupId> 
    109         <artifactId>lily-wal-api</artifactId> 
    110         <version>0.1-dev</version> 
    111       </dependency> 
    112       <dependency> 
    113         <groupId>org.lilycms</groupId> 
    114         <artifactId>lily-wal-impl</artifactId> 
    115         <version>0.1-dev</version> 
    116       </dependency> 
    117       <dependency> 
    118         <groupId>org.lilycms</groupId> 
    119         <artifactId>lily-mq-api</artifactId> 
    120         <version>0.1-dev</version> 
    121       </dependency> 
    122       <dependency> 
    123         <groupId>org.lilycms</groupId> 
    124         <artifactId>lily-mq-impl</artifactId> 
    125         <version>0.1-dev</version> 
    126       </dependency> 
    127       <dependency> 
    128         <groupId>org.lilycms</groupId> 
    129110        <artifactId>lily-rowlog-api</artifactId> 
    130111        <version>0.1-dev</version> 
    … …  
    133114        <groupId>org.lilycms</groupId> 
    134115        <artifactId>lily-rowlog-impl</artifactId> 
     116        <version>0.1-dev</version> 
     117      </dependency> 
     118      <dependency> 
     119        <groupId>org.lilycms</groupId> 
     120        <artifactId>lily-indexer</artifactId> 
     121        <version>0.1-dev</version> 
     122      </dependency> 
     123      <dependency> 
     124        <groupId>org.lilycms</groupId> 
     125        <artifactId>lily-client</artifactId> 
    135126        <version>0.1-dev</version> 
    136127      </dependency> 

  • projects/lily/trunk/process/client/src/main/java/org/lilycms/client/Client.java

    r4048 r4114  
    1515 
    1616import java.io.IOException; 
    17 import java.io.UnsupportedEncodingException; 
    1817import java.net.InetSocketAddress; 
    1918import java.net.SocketAddress; 
    … …  
    6059        AvroConverter remoteConverter = new AvroConverter(); 
    6160        IdGeneratorImpl idGenerator = new IdGeneratorImpl(); 
    62         TypeManager typeManager = new TypeManagerRemoteImpl(parseAddressAndPort(server.repoAddressAndPort), 
     61        TypeManager typeManager = new TypeManagerRemoteImpl(parseAddressAndPort(server.typeManagerAddressAndPort), 
    6362                remoteConverter, idGenerator); 
    64         Repository repository = new RepositoryRemoteImpl(parseAddressAndPort(server.typeManagerAddressAndPort), 
     63        Repository repository = new RepositoryRemoteImpl(parseAddressAndPort(server.repoAddressAndPort), 
    6564                remoteConverter, (TypeManagerRemoteImpl)typeManager, idGenerator); 
    6665        remoteConverter.setRepository(repository); 
    … …  
    6867    } 
    6968 
    70     private SocketAddress parseAddressAndPort(String addressAndPort) { 
     69    private InetSocketAddress parseAddressAndPort(String addressAndPort) { 
    7170        int colonPos = addressAndPort.indexOf(":"); 
    7271        if (colonPos == -1) { 

  • projects/lily/trunk/process/server/README.txt

    r4055 r4114  
    6161 
    6262With standard configuration: 
    63 /path/to/kauri-trunk/kauri.sh 
     63/path/to/kauri-trunk/kauri.sh run -l warn 
     64 
     65The '-l warn' argument makes that all warn-level logging is displayed on the console, which is useful 
     66for local test usage. 
    6467 
    6568With custom configuration: 
    66 /path/to/kauri-trunk/kauri.sh run -c myconf 
     69/path/to/kauri-trunk/kauri.sh run -l warn -c myconf:conf 
    6770 
    6871You can start as many of these processes as you want. By default the server sockets 

  • projects/lily/trunk/process/server/conf/hbase/hbase.xml

    r4048 r4114  
    33       conf:inherit="shallow"> 
    44  <!-- 
    5   <zookeeperQuorum>lat-kind</zookeeperQuorum> 
     5  <zookeeperQuorum>localhost</zookeeperQuorum> 
    66  <zookeeperClientPort>2181</zookeeperClientPort> 
    77  --> 

  • projects/lily/trunk/process/server/conf/kauri/wiring.xml

    r4048 r4114  
    1010    </artifact> 
    1111 
     12    <artifact id="indexer" groupId="org.lilycms" artifactId="lily-indexer-module" version="0.1-dev"> 
     13    </artifact> 
     14 
    1215  </modules> 
    1316 

  • projects/lily/trunk/process/server/conf/repository/repository.xml

    r4048 r4114  
    33            conf:inherit="shallow"> 
    44  <!-- 
    5   <zookeeperConnectString>lat-kind:2181</zookeeperConnectString> 
     5  <zookeeperConnectString>localhost:2181</zookeeperConnectString> 
    66  --> 
     7 
     8  <!-- Filesystem to use to store blobs --> 
     9  <blobFileSystem>hdfs://localhost:9000</blobFileSystem> 
    710</repository> 

  • projects/lily/trunk/process/server/hbase-module/pom.xml

    r4050 r4114  
    2727      <groupId>org.apache.hbase</groupId> 
    2828      <artifactId>hbase</artifactId> 
     29      <exclusions> 
     30        <exclusion> 
     31          <groupId>org.apache.hadoop</groupId> 
     32          <artifactId>avro</artifactId> 
     33        </exclusion> 
     34        <exclusion> 
     35          <!-- also used by solr, but there with a different version of commons-codec, causing 
     36               commons-httpclient to be in Kauri's common classloader but commons-codec not --> 
     37          <groupId>commons-httpclient</groupId> 
     38          <artifactId>commons-httpclient</artifactId> 
     39        </exclusion> 
     40        <exclusion> 
     41          <!-- also used by solr, but there with a different version of commons-codec, causing 
     42               commons-httpclient to be in Kauri's common classloader but commons-codec not --> 
     43          <groupId>commons-codec</groupId> 
     44          <artifactId>commons-codec</artifactId> 
     45        </exclusion> 
     46      </exclusions> 
    2947    </dependency> 
    3048  </dependencies> 

  • projects/lily/trunk/process/server/pom.xml

    r4054 r4114  
    2626    <module>hbase-module</module> 
    2727    <module>repository-module</module> 
     28    <module>indexer-module</module> 
    2829  </modules> 
    2930 

  • projects/lily/trunk/process/server/repository-module/pom.xml

    r4050 r4114  
    2323      <groupId>org.lilycms</groupId> 
    2424      <artifactId>lily-repository-impl</artifactId> 
     25      <exclusions> 
     26        <exclusion> 
     27          <!-- also used by solr, but there with a different version of commons-codec, causing 
     28               commons-httpclient to be in Kauri's common classloader but commons-codec not --> 
     29          <groupId>commons-httpclient</groupId> 
     30          <artifactId>commons-httpclient</artifactId> 
     31        </exclusion> 
     32        <exclusion> 
     33          <!-- also used by solr, but there with a different version of commons-codec, causing 
     34               commons-httpclient to be in Kauri's common classloader but commons-codec not --> 
     35          <groupId>commons-codec</groupId> 
     36          <artifactId>commons-codec</artifactId> 
     37        </exclusion>         
     38      </exclusions> 
    2539    </dependency> 
    2640 

  • projects/lily/trunk/process/server/repository-module/src/main/java/org/lilycms/server/modules/repository/AvroServer.java

    r4048 r4114  
    11package org.lilycms.server.modules.repository; 
    22 
     3import org.apache.avro.ipc.HttpServer; 
    34import org.apache.avro.ipc.Responder; 
    4 import org.apache.avro.ipc.SocketServer; 
    55import org.apache.avro.specific.SpecificResponder; 
    66import org.lilycms.repository.api.Repository; 
    … …  
    1010import javax.annotation.PreDestroy; 
    1111import java.io.IOException; 
    12 import java.net.InetSocketAddress; 
    1312 
    1413public class AvroServer { 
    … …  
    1817    private int typeManagerPort; 
    1918 
    20     private SocketServer repositoryServer; 
    21     private SocketServer typeManagerServer; 
     19    private HttpServer repositoryServer; 
     20    private HttpServer typeManagerServer; 
    2221 
    2322    public AvroServer(String bindAddress, Repository repository, int repositoryPort, int typeManagerPort) { 
    … …  
    3534        AvroRepositoryImpl avroRepository = new AvroRepositoryImpl(repository, avroConverter); 
    3635        Responder repoResponder = new SpecificResponder(AvroRepository.class, avroRepository); 
    37         repositoryServer = new SocketServer(repoResponder, new InetSocketAddress(bindAddress, repositoryPort)); 
     36        repositoryServer = new HttpServer(repoResponder, repositoryPort); 
    3837 
    3938        AvroTypeManagerImpl avroTypeManager = new AvroTypeManagerImpl(repository.getTypeManager(), avroConverter); 
    4039        Responder typeMgrResponder = new SpecificResponder(AvroTypeManager.class, avroTypeManager); 
    41         typeManagerServer = new SocketServer(typeMgrResponder, new InetSocketAddress(bindAddress, typeManagerPort)); 
     40        typeManagerServer = new HttpServer(typeMgrResponder, typeManagerPort); 
    4241    } 
    4342     

  • projects/lily/trunk/process/server/repository-module/src/main/kauri/conf/repository.xml

    r4048 r4114  
    1010  <!-- Port on which to bind the type manager, 0 will bind to an ephemeral port. --> 
    1111  <typeManagerPort>0</typeManagerPort> 
     12 
     13  <!-- Filesystem to use to store blobs --> 
     14  <blobFileSystem>hdfs://localhost:9000</blobFileSystem> 
    1215</repository> 

  • projects/lily/trunk/process/server/repository-module/src/main/kauri/spring/services.xml

    r4048 r4114  
    2323      service="org.lilycms.server.modules.hbase.HBaseConfigurationFactory"/> 
    2424 
     25  <kauri:export-service 
     26      ref="repository" 
     27      service="org.lilycms.repository.api.Repository"/> 
     28 
     29  <kauri:export-service 
     30      ref="messageQueue" 
     31      service="org.lilycms.rowlog.api.RowLog"/> 
     32 
    2533  <bean id="hbaseConf" 
    2634      factory-bean="hbaseConfFactory" 
    … …  
    4149        <constructor-arg> 
    4250          <bean class="java.net.URI"> 
    43             <constructor-arg value="hdfs://lat-kind:9000"/> 
     51            <constructor-arg value="${repository:blobFileSystem}"/> 
    4452          </bean> 
    4553        </constructor-arg> 
    … …  
    5967    <constructor-arg ref="hbaseConf"/> 
    6068  </bean> 
     69 
     70  <bean id="messageQueue" factory-bean="repository" factory-method="getMessageQueue"/> 
    6171 
    6272  <bean id="address" class="org.lilycms.server.modules.repository.AddressResolver"> 

  • projects/lily/trunk/repository/impl/src/main/avro/typeManager.avpr

    r4100 r4114  
    124124         "request": [{"name": "name", "type": "AvroQName"}], 
    125125         "response": "AvroFieldType", 
    126          "errors": ["AvroRepositoryException"] 
     126         "errors": ["AvroFieldTypeNotFoundException", "AvroRepositoryException"] 
    127127     } 
    128128 } 

  • projects/lily/trunk/repository/impl/src/main/java/org/lilycms/repository/impl/AbstractTypeManager.java

    r4100 r4114  
    4848 
    4949    public ValueType getValueType(String primitiveValueTypeName, boolean multivalue, boolean hierarchy) { 
    50         return new ValueTypeImpl(primitiveValueTypes.get(primitiveValueTypeName), multivalue, hierarchy); 
     50        PrimitiveValueType type = primitiveValueTypes.get(primitiveValueTypeName); 
     51        if (type == null) { 
     52            // TODO should probably be another kind of exception, this was just a quick fix to avoid NPE 
     53            throw new IllegalArgumentException("Primitive value type does not exist: " + primitiveValueTypeName); 
     54        } 
     55        return new ValueTypeImpl(type, multivalue, hierarchy); 
    5156    } 
    5257     

  • projects/lily/trunk/repository/impl/src/main/java/org/lilycms/repository/impl/RepositoryRemoteImpl.java

    r4100 r4114  
    44import java.io.InputStream; 
    55import java.io.OutputStream; 
    6 import java.net.SocketAddress; 
     6import java.net.InetSocketAddress; 
     7import java.net.URL; 
    78import java.util.List; 
    89import java.util.Set; 
    … …  
    1213import org.apache.avro.generic.GenericData; 
    1314import org.apache.avro.ipc.AvroRemoteException; 
    14 import org.apache.avro.ipc.SocketTransceiver; 
     15import org.apache.avro.ipc.HttpTransceiver; 
    1516import org.apache.avro.specific.SpecificRequestor; 
    1617import org.apache.avro.util.Utf8; 
    … …  
    3536    private final TypeManager typeManager; 
    3637 
    37     public RepositoryRemoteImpl(SocketAddress address, AvroConverter converter, TypeManagerRemoteImpl typeManager, IdGenerator idGenerator) 
     38    public RepositoryRemoteImpl(InetSocketAddress address, AvroConverter converter, TypeManagerRemoteImpl typeManager, IdGenerator idGenerator) 
    3839            throws IOException { 
    3940        this.converter = converter; 
    … …  
    4142        //TODO idGenerator should not be available or used in the remote implementation 
    4243        this.idGenerator = idGenerator; 
    43         SocketTransceiver client = new SocketTransceiver(address); 
     44        HttpTransceiver client = new HttpTransceiver(new URL("http://" + address.getHostName() + ":" + address.getPort() + "/")); 
    4445 
    4546        repositoryProxy = (AvroRepository) SpecificRequestor.getClient( 

  • projects/lily/trunk/repository/impl/src/main/java/org/lilycms/repository/impl/TypeManagerRemoteImpl.java

    r4100 r4114  
    22 
    33import java.io.IOException; 
    4 import java.net.SocketAddress; 
     4import java.net.InetSocketAddress; 
     5import java.net.URL; 
    56 
    67import org.apache.avro.ipc.AvroRemoteException; 
    7 import org.apache.avro.ipc.SocketTransceiver; 
     8import org.apache.avro.ipc.HttpTransceiver; 
    89import org.apache.avro.specific.SpecificRequestor; 
    910import org.apache.avro.util.Utf8; 
    … …  
    3031    private AvroConverter converter; 
    3132 
    32     public TypeManagerRemoteImpl(SocketAddress address, AvroConverter converter, IdGenerator idGenerator) 
     33    public TypeManagerRemoteImpl(InetSocketAddress address, AvroConverter converter, IdGenerator idGenerator) 
    3334            throws IOException { 
    3435        this.converter = converter; 
    3536        //TODO idGenerator should not be available or used in the remote implementation 
    3637        this.idGenerator = idGenerator; 
    37         SocketTransceiver client = new SocketTransceiver(address); 
     38        HttpTransceiver client = new HttpTransceiver(new URL("http://" + address.getHostName() + ":" + address.getPort() + "/")); 
    3839 
    3940        typeManagerProxy = (AvroTypeManager) SpecificRequestor.getClient(AvroTypeManager.class, client); 
    … …  
    142143        try { 
    143144            return converter.convert(typeManagerProxy.getFieldTypeByName(converter.convert(name))); 
     145        } catch (AvroFieldTypeNotFoundException fieldTypeNotFoundException) { 
     146            throw converter.convert(fieldTypeNotFoundException); 
    144147        } catch (AvroRemoteException remoteException) { 
    145148            // TODO define what we do here 

  • projects/lily/trunk/repository/impl/src/test/java/org/lilycms/repository/impl/test/AvroRepositoryTest.java

    r4100 r4114  
    1919import java.net.InetSocketAddress; 
    2020 
    21 import org.apache.avro.ipc.SocketServer; 
     21import org.apache.avro.ipc.HttpServer; 
    2222import org.apache.avro.specific.SpecificResponder; 
    2323import org.junit.AfterClass; 
    … …  
    5858        AvroConverter serverConverter = new AvroConverter(); 
    5959        serverConverter.setRepository(serverRepository); 
    60         SocketServer repositorySocketServer = new SocketServer(new SpecificResponder(AvroRepository.class, new AvroRepositoryImpl(serverRepository, serverConverter)), 
    61                 new InetSocketAddress(0));  
    62         SocketServer typeManagerSocketServer = new SocketServer(new SpecificResponder(AvroTypeManager.class, new AvroTypeManagerImpl(serverTypeManager, serverConverter)), 
    63                 new InetSocketAddress(0)); 
     60        HttpServer repositoryServer = new HttpServer( 
     61                new SpecificResponder(AvroRepository.class, new AvroRepositoryImpl(serverRepository, serverConverter)), 
     62                0); 
     63        HttpServer typeManagerServer = new HttpServer( 
     64                new SpecificResponder(AvroTypeManager.class, new AvroTypeManagerImpl(serverTypeManager, serverConverter)), 
     65                0); 
    6466        AvroConverter remoteConverter = new AvroConverter(); 
    65         typeManager = new TypeManagerRemoteImpl(new InetSocketAddress(typeManagerSocketServer.getPort()), 
     67        typeManager = new TypeManagerRemoteImpl(new InetSocketAddress(typeManagerServer.getPort()), 
    6668                remoteConverter, idGenerator); 
    67         repository = new RepositoryRemoteImpl(new InetSocketAddress(repositorySocketServer.getPort()), remoteConverter, 
     69        repository = new RepositoryRemoteImpl(new InetSocketAddress(repositoryServer.getPort()), remoteConverter, 
    6870                (TypeManagerRemoteImpl)typeManager, idGenerator); 
    6971        remoteConverter.setRepository(repository); 

  • projects/lily/trunk/repository/impl/src/test/java/org/lilycms/repository/impl/test/AvroTypeManagerFieldTypeTest.java

    r4100 r4114  
    2020import java.net.InetSocketAddress; 
    2121 
    22 import org.apache.avro.ipc.SocketServer; 
     22import org.apache.avro.ipc.HttpServer; 
    2323import org.apache.avro.specific.SpecificResponder; 
    2424import org.junit.After; 
    … …  
    6161        AvroConverter serverConverter = new AvroConverter(); 
    6262        serverConverter.setRepository(serverRepository); 
    63         SocketServer repositorySocketServer = new SocketServer(new SpecificResponder(AvroRepository.class, new AvroRepositoryImpl(serverRepository, serverConverter)), 
    64                 new InetSocketAddress(0));  
    65         SocketServer typeManagerSocketServer = new SocketServer(new SpecificResponder(AvroTypeManager.class, new AvroTypeManagerImpl(serverTypeManager, serverConverter)), 
    66                 new InetSocketAddress(0)); 
     63        HttpServer repositoryServer = new HttpServer( 
     64                new SpecificResponder(AvroRepository.class, new AvroRepositoryImpl(serverRepository, serverConverter)), 
     65                0); 
     66        HttpServer typeManagerServer = new HttpServer( 
     67                new SpecificResponder(AvroTypeManager.class, new AvroTypeManagerImpl(serverTypeManager, serverConverter)), 
     68                0); 
    6769        AvroConverter remoteConverter = new AvroConverter(); 
    68         typeManager = new TypeManagerRemoteImpl(new InetSocketAddress(typeManagerSocketServer.getPort()), 
     70        typeManager = new TypeManagerRemoteImpl(new InetSocketAddress(typeManagerServer.getPort()), 
    6971                remoteConverter, idGenerator); 
    70         Repository repository = new RepositoryRemoteImpl(new InetSocketAddress(repositorySocketServer.getPort()), 
     72        Repository repository = new RepositoryRemoteImpl(new InetSocketAddress(repositoryServer.getPort()), 
    7173                remoteConverter, (TypeManagerRemoteImpl)typeManager, idGenerator); 
    7274        remoteConverter.setRepository(repository); 

  • projects/lily/trunk/repository/impl/src/test/java/org/lilycms/repository/impl/test/AvroTypeManagerRecordTypeTest.java

    r4100 r4114  
    1919import java.net.InetSocketAddress; 
    2020 
    21 import org.apache.avro.ipc.SocketServer; 
     21import org.apache.avro.ipc.HttpServer; 
    2222import org.apache.avro.specific.SpecificResponder; 
    2323import org.junit.*; 
    … …  
    6161        AvroConverter serverConverter = new AvroConverter(); 
    6262        serverConverter.setRepository(serverRepository); 
    63         SocketServer repositorySocketServer = new SocketServer(new SpecificResponder(AvroRepository.class, new AvroRepositoryImpl(serverRepository, serverConverter)), 
    64                 new InetSocketAddress(0));  
    65         SocketServer typeManagerSocketServer = new SocketServer(new SpecificResponder(AvroTypeManager.class, new AvroTypeManagerImpl(serverTypeManager, serverConverter)), 
    66                 new InetSocketAddress(0)); 
     63        HttpServer repositoryServer = new HttpServer( 
     64                new SpecificResponder(AvroRepository.class, new AvroRepositoryImpl(serverRepository, serverConverter)), 
     65                0); 
     66        HttpServer typeManagerServer = new HttpServer( 
     67                new SpecificResponder(AvroTypeManager.class, new AvroTypeManagerImpl(serverTypeManager, serverConverter)), 
     68                0); 
    6769        AvroConverter remoteConverter = new AvroConverter(); 
    68         typeManager = new TypeManagerRemoteImpl(new InetSocketAddress(typeManagerSocketServer.getPort()), 
     70        typeManager = new TypeManagerRemoteImpl(new InetSocketAddress(typeManagerServer.getPort()), 
    6971                remoteConverter, idGenerator); 
    70         Repository repository = new RepositoryRemoteImpl(new InetSocketAddress(repositorySocketServer.getPort()), 
     72        Repository repository = new RepositoryRemoteImpl(new InetSocketAddress(repositoryServer.getPort()), 
    7173                remoteConverter, (TypeManagerRemoteImpl)typeManager, idGenerator); 
    7274        remoteConverter.setRepository(repository); 
Note: See TracChangeset for help on using the changeset viewer.