Archiva

Improve RSS feed generation

Details

  • Type: Improvement Improvement
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: 1.1
  • Fix Version/s: 1.1
  • Component/s: Web Interface
  • Labels:
    None
  • Number of attachments :
    0

Description

  • add check to control file size of generated RSS feeds
  • make sure that the when the client requests the RSS, it gets the right headers to not try and update it if it hasn't changed for performance
  • get the urls of the feeds from the rss feed request

More info in: http://www.nabble.com/Re%3A-svn-commit%3A-r645833---in--archiva-trunk%3A-archiva-jetty-src-main-conf-jetty.xml-archiva-modules-archiva-web-archiva-rss-src-main-java-org-apache-archiva-rss-processor-NewArtifactsRssFeedProcessor.java-td16562163.html

Activity

Hide
Maria Odea Ching added a comment -

Also, clicking the rss feed icon in the repositories page results to a 404 error if the rss feed file does not exist yet.

Show
Maria Odea Ching added a comment - Also, clicking the rss feed icon in the repositories page results to a 404 error if the rss feed file does not exist yet.
Hide
Maria Odea Ching added a comment -

Security:

  • if a user who is subscribed to a feed has been stripped of the necessary permissions or has been deleted, the user should no longer be able to receive updates from the feed
Show
Maria Odea Ching added a comment - Security:
  • if a user who is subscribed to a feed has been stripped of the necessary permissions or has been deleted, the user should no longer be able to receive updates from the feed
Hide
Maria Odea Ching added a comment -

Also, take into account additions via upload (which triggers the consumers, just not the full scan) as specified in the thread referenced above.

Show
Maria Odea Ching added a comment - Also, take into account additions via upload (which triggers the consumers, just not the full scan) as specified in the thread referenced above.
Hide
Maria Odea Ching added a comment -

Fixes in archiva-trunk -r651956:

  • get the contents of the feed from the database
  • updated tests
  • generate the feed without writing to a file

What still needs to be done:

  • rss feed servlet that would serve the feed requests
  • maintain up to which date will be included in the generated feed
  • security
Show
Maria Odea Ching added a comment - Fixes in archiva-trunk -r651956:
  • get the contents of the feed from the database
  • updated tests
  • generate the feed without writing to a file
What still needs to be done:
  • rss feed servlet that would serve the feed requests
  • maintain up to which date will be included in the generated feed
  • security
Hide
Maria Odea Ching added a comment -

Other fixes committed in trunk (-r652043) that were left out from previous commit:

  • the query/constraints for getting the versions of a specific artifact from the db
  • added the 'whenGathered' field in ArchivaArtifactModel
  • added the 'whenGathered' field in beginScan(..) method of RepositoryContentConsumer interface
  • updated all the implementing consumers for the change in the interface
Show
Maria Odea Ching added a comment - Other fixes committed in trunk (-r652043) that were left out from previous commit:
  • the query/constraints for getting the versions of a specific artifact from the db
  • added the 'whenGathered' field in ArchivaArtifactModel
  • added the 'whenGathered' field in beginScan(..) method of RepositoryContentConsumer interface
  • updated all the implementing consumers for the change in the interface
Hide
Maria Odea Ching added a comment -

Fixes in archiva-trunk -r652230:

  • added the servlet that handles the rss feed requests
  • created tests for the servlet
  • updated jsp files for the new rss feed urls
  • removed thrown exception in getSpringConfigLocation() that is being overridden in PlexusInStringTestCase
Show
Maria Odea Ching added a comment - Fixes in archiva-trunk -r652230:
  • added the servlet that handles the rss feed requests
  • created tests for the servlet
  • updated jsp files for the new rss feed urls
  • removed thrown exception in getSpringConfigLocation() that is being overridden in PlexusInStringTestCase
Hide
Maria Odea Ching added a comment -

Fixes in archiva-trunk -r652981:

  • added basic http authentication in rss feed servlet
  • added commons-codec as dependency (used for decoding username and password)
  • updated tests
Show
Maria Odea Ching added a comment - Fixes in archiva-trunk -r652981:
  • added basic http authentication in rss feed servlet
  • added commons-codec as dependency (used for decoding username and password)
  • updated tests
Hide
Maria Odea Ching added a comment -

Fixes in archiva-trunk -r653303:

  • set a limit for the artifacts to be included in the feed (new artifacts in repo)
  • created test for ArtifactsByRepositoryConstraint

The limit is only for the new artifacts in the repository feed, since the per artifact feed content would not be as many as the new artifacts in the repo feed. The constraint for the query would be the artifacts that were gathered within (the current date of the request - 100 days) and (the current date of the request).

Show
Maria Odea Ching added a comment - Fixes in archiva-trunk -r653303:
  • set a limit for the artifacts to be included in the feed (new artifacts in repo)
  • created test for ArtifactsByRepositoryConstraint
The limit is only for the new artifacts in the repository feed, since the per artifact feed content would not be as many as the new artifacts in the repo feed. The constraint for the query would be the artifacts that were gathered within (the current date of the request - 100 days) and (the current date of the request).
Hide
Maria Odea Ching added a comment -

Only thing left to do here is to document it

Show
Maria Odea Ching added a comment - Only thing left to do here is to document it
Hide
Maria Odea Ching added a comment -

The authentication part uses http basic authentication. I tested this with the RSSOwlReader.
Should we also cover http digest authentication?

Show
Maria Odea Ching added a comment - The authentication part uses http basic authentication. I tested this with the RSSOwlReader. Should we also cover http digest authentication?
Hide
Maria Odea Ching added a comment -

Note to myself:

  • test authentication if 'guest' account is enabled for the repo (i don't think this is handled in the feed servlet)
Show
Maria Odea Ching added a comment - Note to myself:
  • test authentication if 'guest' account is enabled for the repo (i don't think this is handled in the feed servlet)
Hide
Brett Porter added a comment -

I gave this a try. In the browser, there were a couple of problems:

  • I hit a 401 immediately. It should have challenged me for the password first?
  • I think it also should have diverted to my netnewswire app - maybe the content-type needs to be sent too?
  • it actually should have skipped these - as you said, it should work with the guest account.

It should be able to share most of this logic with the RepositoryServlet I think.

Show
Brett Porter added a comment - I gave this a try. In the browser, there were a couple of problems:
  • I hit a 401 immediately. It should have challenged me for the password first?
  • I think it also should have diverted to my netnewswire app - maybe the content-type needs to be sent too?
  • it actually should have skipped these - as you said, it should work with the guest account.
It should be able to share most of this logic with the RepositoryServlet I think.
Hide
Maria Odea Ching added a comment -

Changes in -r655027:

  • move servlet authentication (RepoServlet and RssFeedServlet) to archiva-security
Show
Maria Odea Ching added a comment - Changes in -r655027:
  • move servlet authentication (RepoServlet and RssFeedServlet) to archiva-security
Hide
Maria Odea Ching added a comment -

Changes in trunk -r655274:

  • set response header to authenticate if repository is secured
Show
Maria Odea Ching added a comment - Changes in trunk -r655274:
  • set response header to authenticate if repository is secured
Hide
Maria Odea Ching added a comment -

I think everything is done here. I'll leave this open for now until the feature has been tested out by other people aside from me.

Show
Maria Odea Ching added a comment - I think everything is done here. I'll leave this open for now until the feature has been tested out by other people aside from me.
Hide
Brett Porter added a comment -

It looks like it doesn't automatically create the database field if you are upgrading from 1.0.2 (see exception below).

We might need to add a configuration to jpox to have it automatically create new fields.

HTTP ERROR: 500

Error executing JDOQL query "SELECT 'org.apache.maven.archiva.model.ArchivaArtifactModel' AS JPOXMETADATA,THIS.ARTIFACT_ID,THIS.CHECKSUM_MD5,THIS.CHECKSUM_SHA1,THIS.CLASSIFIER,THIS.GROUP_ID,THIS.LAST_MODIFIED,THIS.ORIGIN,THIS.PLATFORM,THIS.REPOSITORY_ID,THIS.FILE_SIZE,THIS.SNAPSHOT_VERSION,THIS.FILE_TYPE,THIS.VERSION,THIS.WHEN_GATHERED AS JPOXORDER0,THIS.WHEN_INDEXED,THIS.WHEN_PROCESSED FROM SA.ARCHIVA_ARTIFACT THIS WHERE THIS.REPOSITORY_ID = ? AND THIS.WHEN_GATHERED >= ? ORDER BY JPOXORDER0" : Column 'THIS.WHEN_GATHERED' is either not in any table in the FROM list or appears within a join specification and is outside the scope of the join specification or appears in a HAVING clause and is not in the GROUP BY list. If this is a CREATE or ALTER TABLE statement then 'THIS.WHEN_GATHERED' is not a column in the target table..
RequestURI=/archiva/rss/rss_feeds

Show
Brett Porter added a comment - It looks like it doesn't automatically create the database field if you are upgrading from 1.0.2 (see exception below). We might need to add a configuration to jpox to have it automatically create new fields. HTTP ERROR: 500 Error executing JDOQL query "SELECT 'org.apache.maven.archiva.model.ArchivaArtifactModel' AS JPOXMETADATA,THIS.ARTIFACT_ID,THIS.CHECKSUM_MD5,THIS.CHECKSUM_SHA1,THIS.CLASSIFIER,THIS.GROUP_ID,THIS.LAST_MODIFIED,THIS.ORIGIN,THIS.PLATFORM,THIS.REPOSITORY_ID,THIS.FILE_SIZE,THIS.SNAPSHOT_VERSION,THIS.FILE_TYPE,THIS.VERSION,THIS.WHEN_GATHERED AS JPOXORDER0,THIS.WHEN_INDEXED,THIS.WHEN_PROCESSED FROM SA.ARCHIVA_ARTIFACT THIS WHERE THIS.REPOSITORY_ID = ? AND THIS.WHEN_GATHERED >= ? ORDER BY JPOXORDER0" : Column 'THIS.WHEN_GATHERED' is either not in any table in the FROM list or appears within a join specification and is outside the scope of the join specification or appears in a HAVING clause and is not in the GROUP BY list. If this is a CREATE or ALTER TABLE statement then 'THIS.WHEN_GATHERED' is not a column in the target table.. RequestURI=/archiva/rss/rss_feeds
Hide
Maria Odea Ching added a comment -

Added autoCreateColumn property in application.xml.
Thanks!

Show
Maria Odea Ching added a comment - Added autoCreateColumn property in application.xml. Thanks!
Hide
Brett Porter added a comment -

because of the timestamps, it shows as updated every time you get it, even if it hasn't changed. I believe the timestamp should just be that of the newest artifact. Also, I think the subject should be just the day, not the time - since the item is meant to be those for that day.

Show
Brett Porter added a comment - because of the timestamps, it shows as updated every time you get it, even if it hasn't changed. I believe the timestamp should just be that of the newest artifact. Also, I think the subject should be just the day, not the time - since the item is meant to be those for that day.
Hide
Brett Porter added a comment -

I also got an NPE on the per-version RSS:
http://localhost:7777/archiva/rss/rss_feeds?groupId=org.apache.maven.scm&artifactId=maven-scm-api

java.lang.NullPointerException
at org.apache.archiva.rss.processor.AbstractArtifactsRssFeedProcessor.processData(AbstractArtifactsRssFeedProcessor.java:50)
at org.apache.archiva.rss.processor.NewVersionsOfArtifactRssFeedProcessor.processNewVersionsOfArtifact(NewVersionsOfArtifactRssFeedProcessor.java:90)
at org.apache.archiva.rss.processor.NewVersionsOfArtifactRssFeedProcessor.process(NewVersionsOfArtifactRssFeedProcessor.java:77)
at org.apache.maven.archiva.web.rss.RssFeedServlet.doGet(RssFeedServlet.java:151)

Show
Brett Porter added a comment - I also got an NPE on the per-version RSS: http://localhost:7777/archiva/rss/rss_feeds?groupId=org.apache.maven.scm&artifactId=maven-scm-api java.lang.NullPointerException at org.apache.archiva.rss.processor.AbstractArtifactsRssFeedProcessor.processData(AbstractArtifactsRssFeedProcessor.java:50) at org.apache.archiva.rss.processor.NewVersionsOfArtifactRssFeedProcessor.processNewVersionsOfArtifact(NewVersionsOfArtifactRssFeedProcessor.java:90) at org.apache.archiva.rss.processor.NewVersionsOfArtifactRssFeedProcessor.process(NewVersionsOfArtifactRssFeedProcessor.java:77) at org.apache.maven.archiva.web.rss.RssFeedServlet.doGet(RssFeedServlet.java:151)
Hide
Maria Odea Ching added a comment -

Thanks for the QA Brett

Fixes in trunk -r656516:
-fix the publish date of the rss entries
-remove if condition in RepositoryContentConsumers.getStartDate()

I've yet to check the NPE you got..

Show
Maria Odea Ching added a comment - Thanks for the QA Brett Fixes in trunk -r656516: -fix the publish date of the rss entries -remove if condition in RepositoryContentConsumers.getStartDate() I've yet to check the NPE you got..
Hide
Maria Odea Ching added a comment -

Fixed NPE in trunk -r656552:

  • query only versions of artifacts with whenGathered != null in ArtifactVersionsConstraint
  • created test for ArtifactVersionsConstraint
Show
Maria Odea Ching added a comment - Fixed NPE in trunk -r656552:
  • query only versions of artifacts with whenGathered != null in ArtifactVersionsConstraint
  • created test for ArtifactVersionsConstraint

People

Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved: