When using hibernate, the JAXB annotation introspector sees hibernate PersistenceSet and not the actual object (JAXB RI has no problems)

I am seeing the same problem, though with a PersistentBag instead of a PersistentSet. Relevant stacktrace:

org.codehaus.jackson.map.JsonMappingException: Unable to marshal: org.hibernate.collection.PersistentBag cannot be cast to Document (through reference chain: Document["documents"])
at org.codehaus.jackson.xc.XmlAdapterJsonSerializer.serialize(XmlAdapterJsonSerializer.java:40)
at org.codehaus.jackson.map.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:446)
at org.codehaus.jackson.map.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:150)
at org.codehaus.jackson.map.ser.BeanSerializer.serialize(BeanSerializer.java:112)
at org.codehaus.jackson.map.ser.StdSerializerProvider._serializeValue(StdSerializerProvider.java:610)
at org.codehaus.jackson.map.ser.StdSerializerProvider.serializeValue(StdSerializerProvider.java:256)
at org.codehaus.jackson.map.ObjectMapper.writeValue(ObjectMapper.java:1604)
at XmlMappingTest.test(XmlMappingTest.java:82)

Property in question:
@XmlElementWrapper(name = "documents")
@XmlElement(name = "document")
@XmlJavaTypeAdapter(DocumentReference.class)
@OneToMany(mappedBy = "parent", cascade = CascadeType.ALL)
@OrderBy("index asc")
public List<Document> getDocuments()

It is quite possible that combination of JAXB handling and Hibernate might not mix; and a few problems have been reported with respect to XmlAdapter in general.
So it might well be a bug, possibly even something easy to fix.

However, Hibernate types require specific workarounds of their own, even aside of JAXB.
So are you using Jackson Hibernate module https://github.com/FasterXML/jackson-module-hibernate?
That is the recommended thing to use when dealing with Hibernate types.

I was not using jackson-module-hibernate, no; I was not aware it was necessary. However, I have now included it and the error remains.

I created a custom mapper:

 
public class CustomObjectMapper extends ObjectMapper {
    public CustomObjectMapper() {
        registerModule(new HibernateModule());
        configure(SerializationConfig.Feature.FAIL_ON_EMPTY_BEANS, false);
        setAnnotationIntrospector(new JaxbAnnotationIntrospector());
    }

    public void setPrettyPrint(boolean prettyPrint) {
        configure(SerializationConfig.Feature.INDENT_OUTPUT, prettyPrint);
    }
}

which uses both the hibernate module and the JAXB Introspector, injected this into my bean (with spring), and got the same error. The stacktrace is also the same. The same entity is successfully mapped by JAXB.

Ok. Would it be possible to write a simple self-contained unit test to reproduce this?

Unit test demonstrating this error.

It turns out this has nothing to do with hibernate, I was able to reproduce the issue with non-persistent classes.

Thanks! Yes, I forgot to mention that it likely isn't Hibernate specific, but rather wrt Jackson JAXB extension handling of adapters.

Ah I see. So the issue is that Adapter should in this case apply to value types of the List, not List itself (and presumably similarly for Maps). Hmmh. Need to think of how to this.

Looks like I can fix this for 2.0, but unfortunately not for 1.x – the problem is that return types from AnnotationIntrospector need to be changed for 'findContentSerializer()' and 'findContentDeserializer()', and this can not be done for patch releases.

Ok. This gets even trickier: apparently it is not true that adapter will ALWAYS apply to value type. Reading through JAXB 2.1 spec, definition suggests something more like:

1. If adapter value type is the same as property type, used for property value directly – can be List or Map, as well as non-structured type
2. If property type is structured (List, Map, array) and parameterized, AND adapter value type matches parameterization (not sure if this applies to key type?), use for parameter type (i.e. value of property; List contents or Map values).

What this means, then, is that rules are elaborate in trying to figure out where adapter applies to Map/List itself, or to values within. I'll do my best to try to map this in some sensible way...

Fixed to the best of my current knowledge; will be in 2.0.0 release of https://github.com/FasterXML/jackson-module-jaxb-annotations